As most of you know by now the repo has been down for a few days due to some security concerns. In this post I will try to explain why I took such measures and more importantly what I did to fix things. Let me start by saying that none of my personal info was leaked. The info shared was that of a repo server I was building. It was in no form secret or private data!
Last Thursday the new 64-bit version of ArmA rolled out, and with it a load of new bugs and mod updates that we had hoped wouldn't be needed. As a result I decided on Friday that I would add a new repo server and set up some form of balancing between the 2. As I was in the process of configuring the new setup I had asked the one of the CM's to test some things for me. During this test 2 things happened. Firstly the CM's got so excited they all decided to start downloading, not what I asked for guys. This caused some frustration as I was trying to do some log tests. Second I noticed someone was actively trying to circumvent the blockades I had put in place during the testing.
Now this in itself I don't mind, if anything I applaud people who test our infrastructure. It helps me make things more resilient and allows you guys to enjoy a safe environment to share your stories in. However! I do not want some kid to go on non FK channels and boast about his escapades. Not only is this a gross breach of trust it is also against everything I stand for myself. What ever he found he should have reported to me, and no one else. Sadly for him he didn't really find anything useful. The info he gathered was something anyone could have found with some simple techniques. Hell I am going to share the same details with you guys right now.
The new repo setup:
In the past we had one repo server that would host a simple web server, this would allow arma3sync to connect via http which it used to download the mods. This box had reached its limits with regards to network speeds and caused painfully slow downloads. To expand this setup I would need a new server and a way to "balance" the load, however due to the amount of data we send out a true load balancer was not an option. Instead I have opted for redirect technique. What does this mean? It means we now have 3 web servers for the repo, one front end and 2 back end servers. In detail:
The front end (a3sync.fuckknows.eu) hosts a small webserver that takes an incoming request and redirects you to another url. It does this by picking a random number between 1 and 4, based on that number it knows what repo to redirect you to. This is not real load balancing, however it does allow us to share load over the 2 repo's. It also allows us to add more servers fairly quickly if we decide to expand in the future.
The back end hosts (repo1.fuckknows.eu & repo2.fuckknows.eu) are web servers that hold the actual repo files. They operate standalone from the front end and allow you to directly connect to should you so desire. They do nothing more but serve files that arma3sync downloads.
Al in all this is not really rocket science, it is a fairly simple setup that allows for easy management and expansion.
What happened last Friday in detail? On Friday evening I changed the a3sync url, instead of pointing directly at repo1 it would now point at the front end server. I had disabled this web server while I did the initial setup and synchronization of the new repo server. During a short period of time I brought it online to perform some tests, in these 5 min tops several people connected and tried to download mods. Once I disabled the front end I expected load on the repo servers to drop again, however that did not happen. A quick look at the log files told me someone was bypassing the front end and when I blocked repo1 that same IP jumped to repo2. I did this testing several times and every time the same client IP popped up. That IP was also using a tool other then arma3sync to download with. I responded by banning that IP from the servers and thought nothing more of it until the screenshot posted by Linnet reached me.
What did I do to prevent this? Even tho I can not make the setup 100% obscured due to the way arma3sync works, I can limit the way's people connect. One of the things I can do is limit the allowed user agents to the web servers, anything other then arma3sync gets a 403 error. I have also increased the monitoring of the log files, repeated attempts to access the repo without arma3sync will get an IP ban. This ban will get propagated to all FK servers in a matter of minutes.
Again I encourage people to try and find issues with our infrastructure, however I do ask you do it in a thoughtful manner. I am not asking you to try and ddos our servers because any monkey can do that for a few bucks. I am asking you to find holes in our security and reporting them to me. If I find anyone exploiting our servers or sharing private info I will not hesitate to report you to the police and your ISP. Needles to say such an action will also result in a ban from our community.
As most of you are aware, the ArmA-Repo we use is currently offline. While Mavy was fixing some issues, we had a community member try and connect to the repo despite being told not to. He ignored our commands, circumvented an IP-ban, searched for a loophole and basicly leaked critical FK-information on a public non-FK channel. This is a very serious breach of privacy, an active defiance of admin/CM orders and completely against everything FK stands for.
As a result of the leak, we have decided to bring down the repo for the time being to fix the issues caused by these actions. This means the repo will be down the entire weekend and no official ArmA-sessions will be played until further notice. For his defiance of the admin/CM team and the leaking of critical information on a public NON-FK channel, the instigator called 'Pretmaker' has been banned from any and all FK-related services.
We take the privacy and security of all FK members very seriously. People trying to circumvent this safety will not be tolerated.
We want to thank Linnet for reporting this grave issue to us. We encourage people who experience similar situations that (could) bring damage to FK to report this as soon as possible to a CM/Admin.
You can read Linnet's explanation in this thread.
We want to take this opportunity to tell People they SHOULD NOT feel like they will be shunned if they were to "snitch" so to speak, because, quite frankly, they're doing the rest of the community a favour. If you are one of those people who start shunning others for openly reporting persons for stuff they shouldn't be doing, then you're honestly no better than the people who are being dickheads and/or breaking/circumventing rules. It shouldn't matter if said person is your friend. If you keep quiet about this, the community as a whole will become stagnant as little things build up and eventually collapse under the combined weight of these issues. If you see someone else has reported your friend for being a dickhead, don't be a dickhead yourself and start shunning that person and/or getting others to shun them.
To wrap it up, don't be afraid to openly report any infractions to the CMs/Admins or even Vets to get them to pass it up to us. The Anonymous Feedback system is still there if you feel afraid of backlash from the community, or at least from a small group of people, but we would much prefer if this became a community where the community itself is not afraid to report people for things that they shouldn't be doing. If we find out that anyone is being a dickhead or shunning people for "snitching", they will be dealt with harshly! We want people to be able to bring their issues out in public where we can have a civil discussion about it. Remember though, if you post an issue, make sure to provide valid points supporting your claims!
Just wanted to give you guys an update on the repo status.
I have pushed the new repo to our servers, yes servers!, the new update is about 7GB due to an issue called RHS. To help deal with the larger amounts of players I have added a second server to "balance" the load between the 2. Note how balance is in quotes, this is because it does not actually balance the load. A server gets picked at random everytime you connect to the repo.
The reason the update is once again several gigabytes is because RHS decided to include an update script in their latest version. Sadly however the servers that contain the updates did not actually get updated. As a result our repo now contained an older version of RHS then before. We never ran into this issue because RHS used to upload their mods to several other sites. However as of 4.2.1 they started using the workshop. Instead I opted to use said script with this disastrous result.
I am still having some difficulty with the new repo, however as soon as those are resolved I will bring the repo back online.
As we feared, we've run into a few snags with the launch of the 64-bit client. As such,
ALL OFFICIAL OPERATIONS ARE SUSPENDED UNTIL FURTHER NOTICE
We will resume regular gameplay once we have our fixes ready. This will mean another repo update, but it will hopefully remain a small one.
Thank you for your understanding.
Updated abramia to 1.8
Updated ace to 3.9
Updated Alive to 22.214.171.1241291
Updated Ares to 0.0.6
Updated CBA to 126.96.36.199227
Updated CUP_Terrains_Core to 1.3
Updated CUP_Terrains_Maps to 1.3
Updated IFA3Lite to v17 - 2017-03-07
Updated Isladuala to 3.66
Updated lingor to 3.8
Updated RHSAFRF to 0.4.2.1
Updated RHSGREF to 0.4.2.1
Updated RHSUSAF to 0.4.2.1
Updated Shacktac to 188.8.131.52
Updated ZADE BOC to 1.1.1
This is a 16GB update, be prepared and start it well in advance!
Whoo-wee guys have we got some exciting this this week!
On Friday Virus is gitting us gud with his second PVP mission. Lock 'n loadaruu!
Two individual instances of training this week - Netheral is hosting some more SL/PltHQ training on Thursday 9th. Be a true RTS gamer.
On Saturday the 11th, Slouchy is taking more FNGs through their paces with his Simple Sessions. Become a bona-fide badass here.
On top of that, Woody is planning to host some Unsung mod Vietnam missions - we have 1 Zeus template & some scripted missions ready to rock! Grab your Steel Pot & your M16 and get ready to do some Zippo raids!
We're still holding our FK Merch design competition - get your designs posted up here!
Starting next week, I will be grabbing some of your awesome posts to feature here too; but for now :