Introduction\n\nWhen we put anything on the internet, we need to be very mindful of what we are opening ourselves up to! Yes, there will be many people eager to make use of the content that we publish, but there are also those out there who will want to test our defenses, as well as make off with any goodies that they can get away with given half a chance. For site owners, it's important to understand what typical usage of sites looks like and be familiar of the peaks and troughs that will naturally occur over the weeks. Website operators also need to be aware when their sites hit technical difficulties, as well as if their sites go down completely as the longer it is unavailable, the longer users aren't getting the content that they crave. Flying blind online isn't advisable!\n\nFortunately, if we implement server side monitoring on our sites, we can get all the information that we need to ensure the smooth running of our resources 😎. Most systems we deploy can log activity externally (e.g. to log files or directly to other systems over a network) at various levels. This means that we can have a rich source of information to understand just how our platforms are generally being operated and adjust our systems where required. For example, if you are running a web-based blog (***ahem), you can see overall daily/weekly/monthly/yearly traffic, how much data is being transmitted, how many hits your profile and articles receive, as well as where the traffic came from using the referrer parameter that comes with every request. You can even see the global region your users are based and the browsers that are used! If you need to understand site status, there are "heartbeat" tools out there that you can use to check your site is online! All of this can be centralised into a central dashboard that you can use to understand your site. All very useful!\n\nYour logs and the understanding of your site's usage can (and SHOULD) be used to understanding malicious activity that likely will exist on your site. Now that site activity can be easily and regularly checked using your fancy dashboard, and you have a STRONG understanding of what your users are doing, you should be using that underlying knowledge to spot activity that doesn't belong. Remember, if you make something publicly available, you don't need to be a high profile candidate to be a target and thus the victim of an attack! NEVER think "it'll never happen to me", as if your site has been online for a day, I can assure you that there have been multiple security scans performed on your site, and maybe even the odd speculative attempt to breach you. If you think otherwise, you clearly aren't reading your logs 😂!!!\n\nSite analytics on a laptop for us to fully understand what's going on behind the scenes\n\nOpening Our Eyes 👁👁\n\nIn order for us to take security seriously, we need to accept that people and/or machines across the internet will try to access anything that we make public due to the ecosystem that we are all a part of. This ecosystem includes (but is not limited to):\n\nUsers that just want utilise one/more resource(s)\n\nSearch engines that crawl the internet for resources, index them and make them easy to find\n\nSecurity researchers who scan the internet for vulnerabilities and responsibly disclose them\n\nMalicious attackers who scan the internet for vulnerabilities and exploit those weaknesses for their own personal/organisational gain\n\nWhen we are monitoring our sites, we will be observing the behaviour of ALL of the identified parties (and those omitted) within the ecosystem, and it's important to be able to discern between the actions that they all take. This is why it's so vital to regularly inspect the logs and understand overall usage. For example, general users (who will likely form most of the traffic) will view the content via the interface and links at a standard pace. Search engine spiders will crawl through links that they find quickly, but they have defined "User-Agent" names (what is used to browse the site) and thus are easy to identify. However security researchers and malicious attackers on the other hand will browse your site in a very different way, enumerating your links and tweaking parameters to see if they can reach places they aren't supposed to. I've written before about this subject, and went into detail regarding a couple of attackers who tried (and failed) to remotely install software on my server and recruit it into their network. In this article I also describe how to protect yourself from such activity.\n\nWhat Hacking Looks Like: Remote Code Execution\n\nThe only way to really spot these kind of actions is to regularly observe your logs and understand your user-base. Then all other activity will stick out like a sore thumb 👍! Even if you want to make use of third-party tools, including an Intrusion Protection System (IDS) and/or a Web Application Firewall (WAF), you still need to add the rules specific for your site to run correctly for your users and break for any such attackers. Neither solution is a drag and drop solution, despite what ANY vendor tells you, and I recommend you have a read of my own early day experience of the EXCELLENT ModSecurity WAF to see what can happen when dropping in a security solution. Fortunately, ModSecurity and the OWASP Core Rule Set I used didn't pretend that it would be easy 🤣! These solutions take time to get right, so be ready to invest if you implement them.\n\nMy WAF Kicked Me Out (ModSecurity)\n\nAttacker tweaking parameters on their keyboard to try and attack websites\n\nDiving a Little Deeper Under the Hood\n\nI'm in a fortunate position, where I have actually written this site hosted at domain heyjournal.com and thus know exactly how I intended for it to be used. This includes how you access the pages available and the underlying API calls which retrieve the data used to populate the site. This means that I am able to spot any obvious requests that shouldn't be there from penetration testers, malicious hackers and/or automated scripts checking out heyjournal for vulnerabilities to exploit. From this, I can say that if you see any 401 (not authenticated) responses it may suggest that guest(s) is(are) trying to access private areas and any 403 (forbidden) responses may suggest that user(s) are trying to access private areas without the required permissions. You can also look out for request methods that you don't use, including standard ones such as CONNECT, HEAD etc, as well as any custom ones that don't belong. Also look out weird byte strings being set to your server(s) as the URL. Standard users will likely not try this, but non-standard vulnerability hunters WILL definitely have a play!\n\nIt is also useful to look out for any non-obvious attempts, like legitimate looking requests that return bad responses (like 404 Not Found) which (might) suggest that someone is taking legitimate links and tweaking the parameters to see if they can get to areas they shouldn't. You should also monitor requests being made without user agent (the browser used to view your content) declarations. EVERY legit client sends this including web browsers, terminals, programming language environments etc, so if it is being omitted someone might be trying to hide who they are and what they are doing from you. For a frame of reference, I've included logfile examples (with IP addresses obscured and referrers changed where needed) taken from the servers which show usage of heyjournal.com. I have split the requests into three different groups:\n\nSuspicious Requests.txt: This file shows a bunch of suspicious requests that seen on heyjournal.com. It includes requesting resources that don't and will never likely exist (404 responses), so where did you get those links from and why do you want those resources? There are requests for DEBUG_SESSION_START for phpstorm which is a remote debugging capability for the Drupal CMS platform. I don't even run Drupal, so why ask?! This is likely to obtain server information from poorly secured instances which shouldn't be shared with anyone bar the developers. There is a possible XSS attack using parameter manipulation between php tags (look for "die(@md5(HelloThinkCMF))"), which is interesting as my platform doesn't use php! There are also bytes being sent to my server (look for anything with \\x) which suggests another protocol is being attempted (my server uses HTTP). If you look at the user agent strings, you'll see that someone tried using a 'AWS Security Scanner' to probe my site for weaknesses!\n\nCrawler Requests.txt: This file shows some of the search engine spider bot requests on heyjournal.com. These came along, crawled the site for content and moved on. Hopefully I get indexed and suggested to as many people as possible 😃.\n\nUser Requests.txt: This file shows seemingly general traffic on heyjournal.com for my profile. Comparing this file to the 'Suspicious Requests.txt' file is night and day, and invaluable experience when trying to combat malicious actions!\n\nSite logging is the equivalent of security cameras monitoring your estate\n\nThese files should demonstrate how the differently a site can be used, and each usage MUST be accommodated. Your standard users expect a smooth, snappy experience with little to no downtime and the search engines spiders need easy to crawl content with high quality links. You also need to balance all of this with taking active measures to secure your infrastructure against the bad people. You don't want to make your site too open otherwise you will be breached, but at the same time you don't want to be too locked down otherwise your intended users won't be able to make use of your site! Monitoring is a great tool to help you with this, enabling you to understand how your site is generally used, and point you in the direction of bad actions that need to be dealt with swiftly.\n\nProtective Actions\n\nSo you've looked at your logs and seen some undesirable behaviour... what do you do now? Here are a few ideas for you to ponder:\n\nDo Nothing: You've seen that activity, but because of the way your site has been developed you are noticing that all such activity is resulting in bad responses, e.g. NOT FOUND, UNAUTHENTICATED, UNAUTHORISED etc., in this case your site is doing everything that it needs to and you don't really need to take further disruptive measures. Of course you need to continually monitor your logs to see if this changes over time, as new vulnerabilities are found every day, attackers get smarter and their tools and techniques evolve all the time.\n\nLog More: Now that you've seen the suspicious activity hitting your webservice(s) and are monitoring the results from this, you need to ask whether or not the information is as clear as you need. Most applications have different logging levels, including information-only, errors, debugging and trace (every little thing). The more information you have, the more insights you can gather (when you can make sense of it). Also, if you are a software developer, I'd recommend going back to your endpoints and adding extra, finely-grained logging information for security failures such as incorrect logins, forbidden access, when private data requests are not found (suggests parameter tampering) etc. Log this to a separate logging category for clarity and monitor this category RELIGIOUSLY.\n\nEnsure that Your Software is Up to Date: Keeping your software patched is vital in the fight against attackers, and includes operating system patches as well as the web framework you are using. As vulnerabilities are found, vendors will release fixes and these fixes NEED to make it to your infrastructure. No excuses! If you get hacked because of a vulnerability fixed 5 years ago because you are running old software, that is on YOU!\n\nDeploy a Web Application Firewall and or Intrusion Detection System: If you are seeing attacks, these will be great in providing that extra level of protection. The rules that you can implement can stop such requests dead in their tracks before they even hit your application, and by logging and monitoring why requests were blocked, you can get an even deeper knowledge in how are being attacked.\n\nBlock IP Addresses: If you are persistently attacked by particular machines, just block them outright so they can no longer send bad requests your way. You can do this at hardware, software and/or web server level, and isn't difficult to implement. The excellent open source Fail2Ban for Linux can do this automatically, and ban such addresses for a predetermined amount of time, or even permanently.\n\nRate Limit Persistent Bursts of Requests: If you are getting bursts of bad requests (maybe the bad guys are trying every different combination of URL or password), implement a rate limit to prevent your services from becoming overwhelmed, or even your accounts being cracked. Be careful and test this out though, as rate limits can affect your users as well, especially if you have a lot of people connecting to your site from the same IP address (e.g. people in the same office).\n\nLock Down the Ports on Your Machine: If you are running an online service, only make the BARE MINIMUM ports available for inbound connections from the internet, which are 443 for all content, and 80 for redirects to 443. Also serve a HSTS header so that user browsers only allow connections over 443. If you are seeing these kind of probes in your logs, you'd better believe that attackers are doing port sweep scans to look for other vulnerable services running on the same server. By locking down your ports to ONLY what is needed to serve content, then these port sweeps will fail.\n\nGet a Penetration Tester: Worried that attackers may find something on your server, or would just like a second opinion? Hire yourself a Penetration Tester to look at your infrastructure for you. You'll likely find their insights invaluable!\n\nSchedule Automated Scans: There are a multitude of free and open source security tools for you to check your site with. I regularly test test instances of heyjournal.com with the Zed Attack Proxy tool available from the OWASP, and there are other tools such as the Metasploit Framework and w3af. A whole heap of these are collected in the Kali Linux distribution, and I'd recommend spinning up an instance and doing some research. It'll really open you eyes 👀!!! Remember, if you use these tools, act responsibly and only use them on infrastructure where you have permission.\n\nZed Attack Proxy (ZAP)\n\nMetasploit Framework\n\nw3af\n\nKali Linux\n\nLady celebrating securing her website after monitoring her site usage \n\nSite Monitoring is Enlightenment\n\nPutting ANYTHING on the internet comes with both benefits as well as risks that we need to know, understand and manage. Website monitoring is a fantastic way of properly understanding how your site is used, and provides very useful insights into how your site might possibly be attacked. I would go as far to say that monitoring is an ESSENTIAL element of running a website, and that because security is everyone's responsibility, we should all take the time to at the very least READ OUR LOGS! We'd all definitely learn a thing or two hahaha 🤣!\n\nTake care and all the best, Si.