jerryhopper 286 Posted June 9, 2021 NOTICE: the bidentify filestructure including the files is available via : https://ftp.armedassault.info/armaholic/ bIdentify ( Pronounced: Bee-Eye-dentify ) A file identification utility-suite for detecting, finding and archiving usergenerated content for Bohemia Interactive games. The bIdentify-client is a commandline tool to scan, detect & identify Addons and Missions archives, folders or .pbo's The bIdentify-server is the database/web-backend for use with the bidentify-client,bidentify-node,bidentify-vault. The bIdentify-node is a Torrent based file-archive that seeds files ( limited to specific size) which are listed in the Bidentify server of your choice. The bIdentify-vault is a Torrent based file-archive that archives and seeds (ALL!) files which are listed in the Bidentify-server. This package is for dedicated librarians who want to preserve and longterm seed all data on the Bidentify server of your choice. Why? Because after 20 years, stuff tends to get lost in many ways. With the recent demise of A-holic it became clear to me that there must be a better and more permanent way of archiving old game-content. Hosting this gamecontent permanently is done by a few site-owners, clans or archivers - like me. But these resources are not eternal, and even i am aware that im not immortal. this project is sole purpose is enabling the community to preserve files with no (or minimal) obligations to any-one. Anyone can host to preserve data, even on very old hardware or slow connections. Inspired by the structure ( and a lot of addon-data ) that i scraped from armaholic, i decided to whip up a 'missing in action' list based on the files i was unable to fetch from armaholic. The script worked, but wasnt very intelligent. it indexed your local directories, and looked for a match on the armaholic mia list - and that's that. What? Once my initial script was used by forummembers, i realised that a text-only database of all addons etc would be a better approach to get as much missing content back, as well as adding new content that wasnt listed on a-holic. But what use is a text-only index of files, if dont have the files, or dont know where to get it? For Arma2 and below, you need to download files from sites, or ftp's. And since the ftp support has been dropped from major browsers, i realised a alternative download protocol is the best way to go if you want to distribute lots & old data. If there was one master-list of all files and its hashes & magnet links, everybody could host (a portion of ) all addons using torrent. In a nutshell, using the client - or hosting a node, vault or server on your old harddisk connected to that 25$ singleboard computer ( a raspberrypi is about 50$) equals supporting the (few) arma-librarians that are left. If you still dont know what you can do? below i try to explain that a bit further.bIdentify-client (Windows .exe/python) (git-repo) Anyone can use the bIdentify-client if they want to clean up their old harddrives - but is not sure if all his old arma-content can be re-downloaded via resources like armaholic. The commandline client is merely a smart indexer, that allows submission of addon-content to the bIdentify network bIdentify-node ( docker/linux ) (git) A clan or enthousiast could be a bIdentify-node, this means hosting (with traffic-caps or timeslots) a portion of the torrents from the masterlist. these node's are volatile. bIdentify-vault ( docker/linux ) A archiver, like myself -or anyone-else would be a bIdentify-vault. this means that ALL data will be hosted via torrent, and is intented to be available 24/7 (with traffic-caps or timeslots)bIdentify-server ( docker/linux ) (git-repo) At least 1 bIdentify-server is needed *( and is hosted by me for now) This is the API backend for the bIdentify-client and can function as a json backend for any website that would like to offer the files in form of magnet-links to the public. it is possible to run multiple bIdentify-server servers, as they all sync with eachother. bIdentify-definitions (git-repo) This is a github-hosted repository which contain the master-and server lists. The master list where all of the bIdentify applications get their serverlists from, are hosted on github, as well as the code. Periodically, the master-index of all files will be uploaded to the github-repository on a yearly basis. the bIdentify-node's will sync content with the available bIdentify-vaults. Vaults and Nodes are the most vital to keep the old file archive preserved. at least 2 bIdentify-vault's and 1 bIdentiy-servers are required to redundantly keep the old-file archives alive. Whats the status? Currently the project is in the start of its development, yet the bIdentify-client is near completion and will be released shortly. 04/6/2021 - started in python, added initial indexing functionality 09/6/2021 - added addon parsing config.cpp, prerelease executable tested. 10/6/2021 - added mission detection, zipped-mission detection and pbo-mission detection 12/6/2021 - initial torrent generation & seeding implemented. the first bidentify-vault is up and running! 13/6/2021 - initial minimalistic web gui in progress (**) 14/6/2021 - missing files added in catalog (**) 15/6/2021 - client-upload is done, server upload facility & torrent generation is almost finished. 24/6/2021 - silent release of the client. (Download here) 05/7/2021 - added pbo inspection to bidentify-node thanks to Mikero ToDo: - Add date & author to files. - Size & file totals per section - simple usage instructions. Where can i download.... Old Armaholic content?! http://bidentify.jerryhopper.com The Bidentify-client?! https://github.com/jerryhopper/bidentify-client/releases Usage : - Download & unpack the zipfile. - open a commandprompt, and navigate to the folder where you unpacked the zip. - type : bidentify.exe -d "D:\directoryToScan" "The past is on your harddisks, the future is in your hands" Hardware A typical bidentify-node or bidentify-vault. this setup costs ~ $50 without the harddisk. ( ~$26 ~$44 ) Obviously, any other singleboard computer can be used, like the RaspberryPi - or any other board with 512MB or more. 25 4 Share this post Link to post Share on other sites
jerryhopper 286 Posted June 9, 2021 Some salvage statistics: The script brought up some interesting statistics of the files i salvaged from armaholic. this index is the baseline for the old files database. arma - Missing: 658 - Salvaged: 1685 (38.31GB) arma2 - Missing: 986 - Salvaged: 1308 (21.42GB) arma2_oa - Missing: 2381 - Salvaged: 0 arma3 - no index. Salvaged: 0 How to Help Obviously, by using the bidentify-client - you automatically help out finding the missing files.... but there is more. in order to keep the files available online there is a need for storage space and internet connectivity. In this day and age many people have 'always-on' connections whether its dsl, cable or even fiber - which means there is a 24/7 internet connection of which the user only uses a small portion of this 24 hours - and it's these people that we are looking for. Any ofp/arma enthousiast willing to share his 'unused' online-time can contribute. This does not mean that your connection will suffer from hosting the files, as the uploadspeed and time-windows are completly customisable. Many of you probably have downloaded a .torrent file before, and this is exactly the same. There are a few options to help 'seeding/mirroring' the torrentfiles. 1. Desktop-computer If you download a file via a torrent, you can choose to 'seed' this file. by seeding the files, you help keep the files online! 2. Dedicated NAS, homeserver or Singleboard computer If you have a NAS or other sort of dedicated computer/server, you can use any torrent-client to seed the files manually, or choose to run a bidentify-node which runs in docker. (*no special technical skills required ) Bandwidth: For availability, not much bandwith is required. Although a upload speed of 50kbps is very slow - it still helps keeping the files available. in general, 100kbps should be fine if there are more than 2 seeders (bidentify-vaults) Another option is to only seed the torrents when you are asleep, or at work. this can be done with the scheduler option in the torrent-software. A slow internetconnection is no excuse for not helping out! Costs: As anyone knows, hosting costs money. If there are enough enthousiasts willing to host some files - the hosting costs for connectivity are zero! But hosting needs hardware. I have made an example calculation for a typical setup needed to run a bidentify-vault A NanoPi NEO3-LTS singleboard computer with 1GB, incl. sdcard, adapter & casing costs roughly 60$ A NanoPi ZeroPi singleboard computer with 512MB, incl. sdcard, adapter & casing costs roughly 45$ A Samsung 870 QVO 1 TB SSD costs roughly $100 The real costs for anyone running a bidentify-vault would be a one-time fee, untill the hardware dies. Power consumption of a SSD and a Singleboard-computer is absurdly low in comparison to a desktop-pc or NAS Obviously, you dont have to buy hardware if you already own a NAS or have a old spare computer with some free harddisk-space So it all boils down to the will to serve some files, and the hardware to do so. The costs are absurdly low, there is no strain on your internet-connectivity due to the excessive configurability of the torrent-software. The key to success is to have at least two vaults running permanently. Who has the will and the resources to become a bidentify-vault!? 2 Share this post Link to post Share on other sites
sarogahtyp 1108 Posted June 9, 2021 Thank you for this great project. I will participate for sure. Maybe as client only but I guess as node or server as well. Depends on needed harddisk space and traffic volume per month. Just waiting for a release. EDIT: Mission accomplished, above Mod identified (I guess): https://www.moddb.com/games/arma-2/addons/tu95-and-tu142-bombers Spoiler 1 Share this post Link to post Share on other sites
jerryhopper 286 Posted June 14, 2021 And then, there was web-userinterface. The details page is still missing, but the download works. Missing files are marked in YELLOW. its now time to finish the commandline-client's upload functionality, and we are good to go! See it live here : http://bidentify.jerryhopper.com/files/arma2/editing/tools 2 Share this post Link to post Share on other sites
Gunter Severloh 4063 Posted June 14, 2021 Jerry keep up the gfood work! idk if it may help but i had compiled all the utilities and tools from Armaholic and on the forum here originally in 2019, last updated March of this year. https://forums.bohemia.net/forums/topic/226269-tools-utilities-compilation-list-for-arma3/ 3 Share this post Link to post Share on other sites
W0lle 1050 Posted June 14, 2021 I think to visually identify an addon we should open a separate thread here. This could also be used to find addons that were not found on A-holic. Similar to the 'Addon Request' thread in the OFP section that is used today to find old addons. This thread should only be used for questions and discussions about bidentify. And that will probably be extensive. Great to see that so many people seem to have files archived and willing to help restoring what is now gone. And even more to see how much time and energy Jerry invests to make the tasks ahead as easy as possible. And I just asked him if he could create an a-holic index.... 👍 3 Share this post Link to post Share on other sites
Bogdan Nasibullin 0 Posted June 15, 2021 18 hours ago, W0lle said: I think to visually identify an addon we should open a separate thread here. 👍 W0lle, I did find it, Rangemaster Pack, I also have files of that addon Share this post Link to post Share on other sites
MarineRecon191 10 Posted June 17, 2021 It genuinely hurts me to see the ARMA 2 OA mods gone... Erased and wiped off the face of the Earth 😞 Share this post Link to post Share on other sites
jerryhopper 286 Posted June 18, 2021 On 6/17/2021 at 7:18 AM, MarineRecon191 said: It genuinely hurts me to see the ARMA 2 OA mods gone... Erased and wiped off the face of the Earth 😞 No! Actually the bidtentify-client spotted about 39gb of missing armaholic in the aa.info ftp. i have been a bit silent, coz there is a web-spot with torrents to all the salvaged files online now. This gives me the time to focus on the scraping again since im very annoyed that there is no date or author , version or required-addons info in the database. it will take not very long after this post, that you will see additional information (date, author, version, author website, required-addons) on the website. The a2oa files will appear in the a2oa section when i run some tests before release of bidentify-client to the public. i aim for a release next week. Scraping the description is undoable, due to the embedded html, links to missing images etc. This information may be scraped from the pbo (in the future). For now, this is the datastructure i managed to scrape from armaholic. { "id":"e3d50d7d-7e58-4f60-afce-e416613f69e5", "author":"Neilreed101", "author_website":"", "requirements":"Everon", "islands":"", "playable_options":"", "version":"3.1", "date":"2007-06-02 16", "filehash":"3592144cd0a4991fa47fbebfc0e0fe53", "filesize":"139567", "signed":"yes, key included", "game":"arma", "armaholicid":"1409", "section":"arma_files_scenarios_spmissions", "name":"Battlefields (@)", "filename":"Reed101_Battlefields.rar", "armaholicpath":"argentina\/scenarios\/sp_missions\/Reed101_Battlefields.rar" } 4 Share this post Link to post Share on other sites
MarineRecon191 10 Posted June 18, 2021 10 hours ago, jerryhopper said: No! Actually the bidtentify-client spotted about 39gb of missing armaholic in the aa.info ftp. i have been a bit silent, coz there is a web-spot with torrents to all the salvaged files online now. This gives me the time to focus on the scraping again since im very annoyed that there is no date or author , version or required-addons info in the database. it will take not very long after this post, that you will see additional information (date, author, version, author website, required-addons) on the website. The a2oa files will appear in the a2oa section when i run some tests before release of bidentify-client to the public. i aim for a release next week. Scraping the description is undoable, due to the embedded html, links to missing images etc. This information may be scraped from the pbo (in the future). For now, this is the datastructure i managed to scrape from armaholic. { "id":"e3d50d7d-7e58-4f60-afce-e416613f69e5", "author":"Neilreed101", "author_website":"", "requirements":"Everon", "islands":"", "playable_options":"", "version":"3.1", "date":"2007-06-02 16", "filehash":"3592144cd0a4991fa47fbebfc0e0fe53", "filesize":"139567", "signed":"yes, key included", "game":"arma", "armaholicid":"1409", "section":"arma_files_scenarios_spmissions", "name":"Battlefields (@)", "filename":"Reed101_Battlefields.rar", "armaholicpath":"argentina\/scenarios\/sp_missions\/Reed101_Battlefields.rar" } That's great to hear, I can't thank you enough for doing this for us! Share this post Link to post Share on other sites
W0lle 1050 Posted June 18, 2021 👆 That. Still a lot of usable information in there. Author, game, name, file name. That's something to work with. Share this post Link to post Share on other sites
jerryhopper 286 Posted June 22, 2021 Despite my attempts to release the client today, i didnt make it. At this very moment i have a client now indexing and uploading missingfiles.. a lot of arma2 OA stuff, and quite some arma1-weapons addons are being added, which means that the client is almost ready for use. I encountered some absured large addons like CUPS (10GB) that made the system crash, which means i have to delay the release. 10 Share this post Link to post Share on other sites
W0lle 1050 Posted June 23, 2021 Still progress made. A lot. Was very pleased to see the client in action today. Good job Jerry on bringing the lost items back. 2 Share this post Link to post Share on other sites
krzychuzokecia 719 Posted June 24, 2021 On 6/23/2021 at 12:51 AM, jerryhopper said: absured large addons Speaking of file-size - I'm on a rather slow connection, does the client allow to set some priority on uploading files? So I could start with small ones, and get to the bigguns later. Also, what's the further plan? I mean - having all the files backed up seems like only part of the job. To make the archive usable for people, the files should be accompanied by some sort of description, maybe simple tags system? I know you plan some sort of config lookup, but classnames are often not enough. Share this post Link to post Share on other sites
jerryhopper 286 Posted June 24, 2021 1 minute ago, krzychuzokecia said: Speaking of file-size - I'm on a rather slow connection, does the client allow to set some priority on uploading files? So I could start with small ones, and get to the bigguns later. Also, what's the further plan? I mean - having all the files backed up seems like only part of the job. To make the archive usable for people, the files should be accompanied by some sort of description, maybe simple tags system? I know you plan some sort of config lookup, but classnames are often not enough. The bidentify client doenst prioritize on size. it skips files bigger than 900MB (What you can do, is make bidentify scan the folder with smaller stuff first) Also, what's the further plan? I will first finish the system. in other words : the bidentify-node/vault should be 100% working - as in: Everyone should be able to run a dedicated vault or node. The 'web frontend' should be a deployable container. the 'absolute' file/hash list should be periodically (yearly?) backed up to a git-repository. The idea is that anyone who is a bit tech-savy should be able to deploy the bidentify suite with ease, (in case i somehow die in the short future) and that the files should be 'kept' with a few trusted archivers. i do understand the need for better search-options, and i got that covered. i have anticipated on this, and i will implement a redis-search module which allows you to search on all kind of properties in the near future. Also, i want to 'analyze' pbo's to extract content and add this to the database. adding 'users' and 'posting/commenting' functionality is not on my "to be implemented list." Another very important part is not only the Site & client - but people who are willing to share a bit of their internet connection, and host a node or vault. Without these - the project doesnt have a future. I can produce a OS-image with the vault-software preinstalled for cheap single-board computers, like raspberry pi or others. Dont forget the initial idea is to keep the files available to the public - not to rebuild armaholic. But if anyone feels like interacting with the data in the bidentify database - and build some website on top of it - drop me a message. there is an API - we'll figure it out! Share this post Link to post Share on other sites
krzychuzokecia 719 Posted June 25, 2021 Is the bIdentify client supposed to usable already? I can't get it to work - running as administrator, bIdentify added to anti-vir exemptions etc. I'm using the syntax from first post/github release page, but it doesn't seem to work. Matter of fact, even the help commands don't work. Share this post Link to post Share on other sites
W0lle 1050 Posted June 25, 2021 Definitely works, uploaded a bunch of files before. Correct command is: bidentify scan -d d:\temp\arma 1 Share this post Link to post Share on other sites
krzychuzokecia 719 Posted June 25, 2021 Yay, that syntax worked! I uploaded 3 megs of missing A1 files, now going through 5 gigs of A2 ones - my ISP will hate me for that, but it's okay, I'll hate them back! :D Edit: @jerryhopper currently client responds "Server full, please wait 15 sec" - I think it started about 20-30 minutes ago. 2 1 Share this post Link to post Share on other sites
jerryhopper 286 Posted June 25, 2021 3 hours ago, krzychuzokecia said: Yay, that syntax worked! I uploaded 3 megs of missing A1 files, now going through 5 gigs of A2 ones - my ISP will hate me for that, but it's okay, I'll hate them back! 😄 Edit: @jerryhopper currently client responds "Server full, please wait 15 sec" - I think it started about 20-30 minutes ago. Hey, im sorry to hear you are brutally interupted by a full server! can you please try again tomorrow? I am upgrading the 'vault' to a RaspberryPi4 (with 4gb ram) which has more usb-ports, and allows booting from ssd. This means the storage-vault (with torrents) is unavailable, since im the only one running one. So, despite the client error-message that says 'server full', its actually a good thing - as everything works as its supposed to 🙂 PS: i see that i made a mistake in the commandline instruction, i will fix that asap... thanks W0lle for the correct command! 1 Share this post Link to post Share on other sites
krzychuzokecia 719 Posted June 25, 2021 8 minutes ago, jerryhopper said: can you please try again tomorrow? Of course! It makes me so damn happy to see it working, you're saving the community once again Jerry! Thank you so much! Share this post Link to post Share on other sites
jerryhopper 286 Posted June 29, 2021 So, Today i got a reminder from the 'GoFundMe' project (*https://www.gofundme.com/f/free-armaholic) I'm not entirely sure what to do with it, as it seems clear that armaholic cannot be 'bought' back. I have overthought the idea of using the funds to buy another HDD, and use that for bidentify. But somehow this doesn't feel 'right', Therefore i'm expiring / refunding the supporters. Although i cannot find any refund-option in the go-fundme website, when the campaign expires the funds are automatically refunded. Last but not least, i'd like to thank Joonas Sariola, Dennis GamerOnkel, BBFrost and 'Anonymous' for their generous donations. 5 Share this post Link to post Share on other sites
lukin86 24 Posted July 1, 2021 I have some total mod arma 2 but also arma 1 that I could dl before closing. I could send to the server. Share this post Link to post Share on other sites
sarogahtyp 1108 Posted July 1, 2021 25 minutes ago, lukin86 said: I have some total mod arma 2 but also arma 1 that I could dl before closing. I could send to the server. You can just use the client for uploading: On 6/25/2021 at 5:45 PM, W0lle said: Definitely works, uploaded a bunch of files before. Correct command is: bidentify scan -d d:\temp\arma Share this post Link to post Share on other sites
krzychuzokecia 719 Posted July 1, 2021 When trying to upload a particular file (@DIH_Africa2.zip 531MB), I'm getting the following: Traceback (most recent call last): File "urllib3\connectionpool.py", line 706, in urlopen File "urllib3\connectionpool.py", line 394, in _make_request File "urllib3\connection.py", line 234, in request File "http\client.py", line 1252, in request File "http\client.py", line 1298, in _send_request File "http\client.py", line 1247, in endheaders File "http\client.py", line 1065, in _send_output File "http\client.py", line 987, in send ConnectionResetError: [WinError 10054] Istniejące połączenie zostało gwałtownie zamknięte przez zdalnego hosta During handling of the above exception, another exception occurred: Traceback (most recent call last): File "requests\adapters.py", line 449, in send File "urllib3\connectionpool.py", line 756, in urlopen File "urllib3\util\retry.py", line 532, in increment File "urllib3\packages\six.py", line 769, in reraise File "urllib3\connectionpool.py", line 706, in urlopen File "urllib3\connectionpool.py", line 394, in _make_request File "urllib3\connection.py", line 234, in request File "http\client.py", line 1252, in request File "http\client.py", line 1298, in _send_request File "http\client.py", line 1247, in endheaders File "http\client.py", line 1065, in _send_output File "http\client.py", line 987, in send urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(1 0054, 'Istniejące połączenie zostało gwałtownie zamknięte przez zdalnego hosta', None, 10054, None)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "bidentify.py", line 47, in <module> File "bidentify.py", line 40, in main File "lib\bidentify.py", line 46, in start File "lib\bidentify.py", line 96, in doScan File "lib\scandir.py", line 68, in scandir File "lib\scandir.py", line 134, in submitFiles File "lib\scandir.py", line 168, in submitItem File "requests\sessions.py", line 590, in post File "requests\sessions.py", line 542, in request File "requests\sessions.py", line 655, in send File "requests\adapters.py", line 498, in send requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetErro r(10054, 'Istniejące połączenie zostało gwałtownie zamknięte przez zdalnego host a', None, 10054, None)) [9140] Failed to execute script bidentify Error 10054 description is "An existing connection was forcibly closed by the remote host". Happened three times today, I'm not sure if it's the filesize that's problem, or the file itself. At the moment I'm uploading the rest of stuff that I have. Share this post Link to post Share on other sites
W0lle 1050 Posted July 1, 2021 I have that too every here and then. But good to know I'm not the only one and it's not my lame connection. So... let's blame Jerry then. 🤣 1 Share this post Link to post Share on other sites
