Introduction\n\nIt's very useful to make our resources available online for specific persons to access. I mean, who knows when they will come in handy right 🤷♂️?! Within the increasingly connected world, there is also a growing expectation to be able to access what we want, where we want whenever we need to, which explains the drive to put more of our electronic items online (hopefully) to our overall benefit 😎.\n\nUnfortunately, there aren't only positive elements to putting our resources online that we have to consider. By making our files freely available to us wherever and whenever, we are also providing opportunities for malicious persons to steal our electronic data. As a preventative measure, it's important that we inspect any file requests that we receive to ensure that we serve the correct resources to those with the right identity, permissions and access. Anything else could be a data breach/leak 😱!\n\nHere I'm going to discuss some of the techniques developers can use to protect files that we host online!\n\nLady choosing which files that she needs\n\nServing Our Files\n\nWhen serving our files across the internet, we REALLY need to be careful. Yes it's convenient and very useful to always be connected, and during most occasions everything will be fine and dandy 😁. However, a simple file-server application designed to serve a subset of files could potentially be weaponised into a tool that serves all the files (including secret and security-critical) on the host machine if certain vulnerabilities are present. If we don't effectively check and validate client inputs sent to our server, then we could be in for a TERRIBLY rough ride!\n\nOWASP: Path Traversal\n\nPortSwigger: Insecure Direct Object References (IDOR)\n\nOWASP: Top Ten 2017: A5:2017-Broken Access Control\n\nVulnerabilities such as 'Path Traversal' and 'Insecure Direct Object Reference' are not to be scoffed at, and if your platform is vulnerable to these kind of nasties, they can allow miscreants to manipulate your resource links, potentially allowing them to bypass measures that you have put in place and subsequently break your access control 😱! Files intended "For Your Eyes" only can now be enumerated, pilfered and subsequently cause you VERY sleepless nights unfortunately.\n\nEye keeping a lookout for suspicious behaviour\n\nFortunately, these issues (and other critical ones) are very well known and have been documented by very knowledgeable people (Top 10 Vulnerabilities attached, and link below), meaning that we have a fighting chance to protect ourselves 💪! Such lessons learned have been compiled into handy guides for us to easily base our software development around, enabling us to learn from the mistakes of the past and keep the bad people out 😁. There is even guidance which specifically outlines best practice for file uploads (link below), and a comprehensive list of overall Secure Coding Practices (attached).\n\nOWASP: File Upload Cheat Sheet\n\nOWASP: Top 10 Vulnerabilities\n\nBeing Safe and Secure\n\nIn order to safeguard our online resources from malicious access, we need to think about how we store files and what we permit when our users make requests. If we take into consideration the file storage guidelines, the other secure coding practices as well as preempt the various techniques that attackers may use to bypass our security, we can go a long way in protecting our digital assets from theft and/or leakage. I've had a think about some of the ways we can protect ourselves, and distilled them into clear Requirements that any system can make use of:\n\nThe file base name shall be a v4 UUID. [Makes validation easy as all files have a fixed pattern containing only safe characters using a format where the probability of collisions (two files with the same base name) is EXTREMELY low]\n\nThe user id shall have a predefined, fixed-length format. [Makes validation easy as all user IDs have a fixed pattern containing only safe characters using a format where the probability of collisions is EXTREMELY low]\n\nOnly relative file paths shall be used which are mapped to a gated file context on the server SEPARATE to the application running. [Prevents attackers from potentially reading secrets from application configuration files, as well as escaping the application context and reading data from the rest of the server]\n\nFile paths shall have a predefined list of prefixes that accurately describe the file type(s). [Prevents Path Traversal as ANY path provided outside of this controlled whitelist will result in error]\n\nThe format of a file path shall be userId/prefix/fileName. [Makes validation easy as path structure never changes, and prevents Path Traversal and URL manipulation as any input outside of this fixed structure will be rejected]\n\nFile extensions shall be limited only to predefined media types that shall be served. [Ensures that only media intended to returned can be returned and that calls made to other files (e.g. a private key .pem file or a sensitive configuration .conf file) can be easily rejected]\n\nFile range requests (result in partial content along with 206 response) should have clearly defined limits (e.g. response <= MAX_RESPONSE_SIZE). [Prevent server from returning crazy sizes that might stretch bandwidth, max out server resources etc.]\n\nFile range requests (result in partial content along with 206 response) should have strict input validation (e.g. start >= 0, start < end, end < size etc.). [Prevents attackers from manipulating returned bytes and potentially reading areas of storage they aren't supposed to]\n\nAll bad file requests should have the same error message. [Limit attackers learning too much about our system(s) from our error responses]\n\nAll file access errors should be centrally logged along with precise reasoning. [Makes it easy to monitor usage and detect any malicious behaviour]\n\nArrows ➡️ guiding us to a more secure file server\n\nTo read more about the UUIDs mentioned for the filenames, as well as get more information on file range requests, please see the resources included below.\n\nRFC4122: A Universally Unique IDentifier (UUID) URN Namespace\n\nMozilla Developer Network: Range Requests\n\nIt Pays to be Paranoid 😂\n\nAdherence to the above requirements will go a long way in preventing what an attacker can do with a system as file retrieval had been made incredibly strict, whilst still remaining simple to use. On top of this, the logging and shipping of file usage, along with precise file access errors into a central system will permit accurate system monitoring and alert you to any "questionable" behaviour. Such monitoring could be displayed on a dashboard for easy viewing both for historical and real-time data 🤓. I've included an example dashboard image from a working solution that I built below, which shows the filetypes received, errors over time, the percentage of errors during the time period and a more detailed breakdown of each error.\n\nFile access dashboard showing filetypes received, errors over time, the percentage of errors during the time period and a more detailed breakdown of each error\n\nMonitoring site usage is as important as blocking bad requests as it ensures that you fully understand how your resources are actually being used, so ensure that you give your system the care and attention that it deserves 😎. I have written an article about this previously and thoroughly recommend that you check it out!\n\nHeyJournal: Monitor Your Things, Check Your Logs!\n\nProtecting Our Online Resources\n\nPutting our files on a public network comes with many benefits, along with multiple risks that we have to consider and manage accordingly. Fortunately, many of the dangers have been documented and solved, meaning that we can design resilient systems which can prevent bad behaviour. It is also important that we record (what we perceive as) bad-faith access to a central system so that we can easily monitor our platforms and be aware of our system usage. By doing this, we can all stand to benefit from data that has been available to us, without the additional stress!\n\nTake care and all the best, Si.