I’m interested in a lot of stuff when it comes to tech, but I don’t think anything entertains me more then trying to figure out how stuff works. From time to time I come across things that are just too silly to ignore.
I remember a few years back while uncovering that all of GET’s routers were accessible directly from the web and they didn’t even change the default login. Even after notifying them they didn’t care, but luckily they patched it a few weeks later. Anyway.. today I have another (my first on my blog!) fail.. which targets at least three mayor newspapers in Norway.
Newspapers want to protect their articles, which is somewhat understandable. The fun part is when they want to protect the information, but don’t want to sacrificing SEO or Search Engine Optimization. So how are you supposed to solve both issues? If you require the users to be logged in, search engines like Google or Bing won’t be able to access and index your content, thus ranking your site lower. Two of the local newspaper where I live protect their content by checking the user agent of your browser. The sites in question are Grimstad Adressetidende and Agderposten.
Browser user agent?
Each time you visit a website your browser says something like “Hi, I’m Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36”. However, when Google visits a site, it says just “Hi, I’m Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”. You might already have guessed it… why not just tell your browser to say it’s Google? Yes you can! There are a lot of addons which will do that for you. If you are using FireFox you could use UAControl. Just specify ” Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ” and you’ll have full access to any article on those sites.
Checking your referral
Another case I had was that I got a link from a friend on Facebook to an article published on the newspaper Fædrelandsvennen. The strange thing was that the person sending me the link you read the article, while I only got to a page asking me to pay for access. While discussing this for a few minutes, it came to my attention that the site was reading my browsers referrer. In other words the site is configured to check where I come from when visiting the site. In this case the site is configured to allow anyone coming from Facebook to read their articles. Once you know this, it was really easy to find a addon that allowed me to set my browser referrer depending on the site I visit. If you are using FireFox you may RefControl. Now you can read every single article without limitations.
Security through obscurity is just silly and there’s no benefit from doing things this way. Even though I understand the reason why these newspapers chose this solution, I think requiring authentication would be a much better way of doing it. My slogan is “makes it decent or not at all” and in this case it would without doubt be… not at all. Well, at least this was a fun thing to be made aware of or did you already know?
I’m sure this also works on other sites. Give it a try and let me know!