Menu

Articles & Blogs

Penetration Testing: Search Engine based Reconnaissance

October 20, 2022 | By Accorian
ISO 22301 BUSINESS CONTINUITY MANAGEMENT SYSTEM

Written by Vivek Jaiswal II 

Reconnaissance is an essential phase in Penetration Testing, before actively testing targets for vulnerabilities.

It helps you widen the scope & attack surface and helps uncover potential vulnerabilities. There are already multiple open-source and proprietary automated tools available in the market to perform reconnaissance or scan any host/application for vulnerabilities, while penetration testing. However, the manual and professional approach is what gives you the actual understanding of the backend technology, it’s workflow, and helps you uncover potential vulnerabilities.

A basic reconnaissance includes:

  • Subdomain Enumeration
  • Directory Enumeration
  • Port Scanning
  • Search Engine based recon
  • Github recon
  • Shodan recon
  • Enumerating backend technologies
  • WayBack History
  • and so on

 

In this article, I will demonstrate how a simple Search Engine based Reconnaissance helped me identify a potential security vulnerability that leads to dumping the entire database – SQLi

While I was recently working on an External Network Penetration Testing project, as usual, I started with the basic reconnaissance approach.

Now, for a Network Penetration Testing activity, I started with the basic port scan and services enumeration. Once the scans were complete, I found the 80/TCP port open which is an HTTP webpage. I then quickly visited the site and found that it did not have any feature or functionality and was only a static error page.

Search Engine Based Reconnaissance

After this, I started performing some directory brute forcing using a common wordlist of directories. You can find the payload list here.

I couldn’t find any valid directory or any entry point, on trying the common directory wordlist. I then proceeded with another reconnaissance approach which was the Search Engine Based Information Discovery.

Tip: It is always better to use custom directory names wherever possible, as they are difficult to guess and brute force.

Here comes the interesting part

First, what is Search Engine based discovery or reconnaissance in penetration testing?

Search Engine based reconnaissance in simple terms is a method to extract all the information which are publicly available on the internet in the databases of various search engine(s).

Basically, all search engines work in an automated fashion where they use software known as web crawlers that explore the web regularly to find pages to add to their indexes. In fact, the vast majority of pages listed in our results aren’t manually submitted for inclusion but are found and added automatically when the web crawlers explore the web.

Coming back to our scenario:

While I was trying to enumerate via Search Engine discovery, looking for information publicly disclosed over the Internet, I came across a very interesting directory. In this article, I will call it “/unique_directory”.

Search Engine based Reconnaissance

I quickly accessed the URL (http://example.com/unique_directory) and found a simple login page.

Search Engine based Reconnaissance

I then started to poke the login page to find weaknesses that I could leverage and, in the process, I sent a single quote (‘) as the username. The server responded with an error page, and the error message indicated that there was an unclosed quotation mark, as the addition of the single quote made the backend query syntactically incorrect.

this behavior and the error message from the server,is an indication of a possible SQLi.

After numerous attempts of carefully crafting and recrafting payloads, it was observed that the servers revealed the backend database information in the error message, which confirmed the presence of a SQLi vulnerability and also the database server used in the backend.

Search Engine based Reconnaissance

Payload used: ‘) and (select CASE WHEN (substring(@@version,1,50))=1 THEN 1 ELSE 0 END )=1 and (‘1’=’1

Explanation of the payload:

The CASE statement goes through conditions and returns a value when the first condition is met (similar to an if-then-else statement). So, once a condition is true, it will stop reading and return the result. If no conditions are true, it returns the value in the ELSE clause. You can find details of the syntax here.

What are some of the preventive measures one should always consider overcoming such scenarios?

As part of your vulnerability or security management strategy, you need to continuously identify and remediate vulnerabilities in every small, medium, and large business application, because there is never a one-time or one-patch solution for vulnerabilities. The business applications, hosts, assets, and every single piece of information which are posted online need to be audited and monitored in a regular and timely fashion.

Hence, the best recommendation to overcome such scenarios is to carefully consider the sensitivity of design and configuration information before it is posted online and to periodically scan them.

Sometimes, the robots.txt files also work efficiently in preventing sensitive pages, directories, or information from being automatically crawled by search engines and getting stored in their databases.

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests. However, it is still not a fool proof solution.

vulnerability or security management strategy

One can also Block Search indexing with noindex: A noindex meta tag or header can be added to the HTTP response to prevent a website or other resource from showing up in Google search. Regardless of whether other websites link to it, the page will be completely removed from Google Search results when Googlebot crawls it again and notices the tag or header.

Similarly, we can also temporarily block search results from your site or manage safe search filtering.

Details on this can be found here.

Conclusion

Every piece of information available on the internet could be sensitive and can be a potential entry point for attackers, which could later be escalated to dump the entire database.

Even if you are unintentionally exposing the sensitive information, it will still be crawled through and will be indexed into the search engine Databases, which an attacker can easily extract and enumerate via search engine based discovery or Google dorking.


Useful Tip:  While performing search engine based reconnaissance, do not limit testing to just one search engine provider, as different search engines may generate different results. Search engine results can vary in a few ways, depending on when the engine last crawled content, and the algorithm the engine uses to determine relevant pages.

Every small, medium, and large application, host, API, or any piece of information posted online should be thoroughly examined. It is highly recommended to review the sensitivity of the online information on current designs and configurations, on a regular basis.

Recent Blog

Article

Adobe's Common Controls Framework of Industry-acclaimed security standards

Today’s world is an ever-changing scenario with changes to the technology sector happening more frequently than ever due to emerging technologies. The case is quite similar in the field of Cyber Security. There are a few industry-acclaimed cybersecurity standards for governing the processes and execution of these standards. These standards are usually built upon a framework of control objectives that need to be implemented by the organizations to comply with these standards. Compliance is measured in terms of control objectives meeting the compliance criteria and also other regulatory and statutory criteria. Since most of these Cybersecurity standards speak of similar control objectives or lay emphasis on similar control areas, it is advisable to have the ‘Adobe’s Common Control Framework’, which means that if we are able to comply with a single requirement from a particular framework, in theory, we should be able to use the adherence of that requirement for ALL the similar frameworks. There are several approaches to achieving this Adobe's Common Controls Framework both in theory and in practice and will be discussed in detail later on in this article. The most relevant security and privacy frameworks are ISO 27001, NIST, PCIDSS, GDPR, SOC Type 2. There is a significant overlap of controls contained in these standards as all of these standards primarily deal with one requirement which is the protection of data. Protection of information from unauthorized disclosure, compromise, and theft forms the backbone or the building blocks of an Adobe's Common Control Framework. This leverages the fact that similar controls or that the essence of the controls is the same across standards and can be used to gauge the adherence or compliance of an organization to the standard. In actual execution, while gauging the compliance of an organization, the Adobe's Common Controls Framework is not only holistic but can reduce the effort and cost otherwise required by the organization to comply with individual standards. There are two methods of developing the Adobe's Common Control Framework for an organization and there are very subtle differences between the two methods. They are Controls harmonization and Controls Mapping. Controls Harmonization: Harmonization is the creation of a brand-new control language set from several source languages of standards taking into consideration content & context. In theory, the intent and meaning of the words and sentences remain intact, but the language and actual words of the individual standards have been changed with a new harmonized meaning defined. To achieve the usage of a single language as an industry, globally, it would have significant benefits and it would be the most efficient way to operate not only as security professionals but also as humans. Adobe’s Common Control Framework is an example of this type of construct. The benefits of having a single operating language can truly be amazing in terms of effort reduction and cost reduction. Control Mapping: Today, most brilliant and forward-thinking security professionals are using the control mapping method. The main idea behind this method is to keep the original language intact as much as possible while mapping and matching the intent and meaning of each sentence and word. This is the most practical and realistic approach because this is how humans fundamentally interact with each other globally. One can see this working in real life where two different languages are being spoken by individuals and kept mainly intact, but an interpreter or linguist is translating between the two — the map is developed in the mind of the linguist. Some real-life examples of mapping for cybersecurity frameworks can be seen in HITRUST Framework, Cloud Security Alliance Framework, and even the U.S. Government formally...

View More

Article

Risk Management Framework – Managing & Measuring what matters

A risk management program allows you to manage overall information security risk.  It is an approach to identify, quantify, mitigate, and monitor risks.  The reason to look at risk in a comprehensive manner is to make sure no one area is getting too much attention or, too little. Frameworks also help you identify the bigger elements of risk that need review and mitigation. Additionally, it help you to prioritize the treatment of certain risks. The adherence to risk frameworks also helps give your customers comfort that a standard risk management is in place and hence, reduce your audit burden.  Typically, a Risk Management program comprises of the following phases: Risk identification Risk analysis Risk evaluation Risk treatment Risk Monitoring A good risk management framework will have the following characteristics: Comprehensive in types of risks it covers Practical for an organization to implement Updated with current real-world risks Based on controls that can be reviewed and audited Reliable so that your vendors and customers can accept it There are many risk management frameworks that one can choose from and it important to understand the advantages of each. Common risk management frameworks include: NIST CSF SOC 2 ISO 27001 HITRUST NIST (National Institute of Standards and Technology) framework is a comprehensive cybersecurity framework (CSF).  It is very widely accepted across different company sizes and business domains. Most cyber-security professionals are well aware of this framework as it is readily available.  Although widely available and very popular there is no certified third-party audit mechanism. Hence, it can only be self-assessed.   SOC 2 Type 2 is an internal controls report based on the scope you define.  It is widely used in the United States to show the maturity of your controls.  A CPA firm that is part of the American Institute of CPAs (AICPA) conducts the audit & issues an assessment report.  The AICPA does not audit/review the assessment for completeness or quality.  HITRUST CSF is a framework that came leverages NIST, SOC, and ISO along with others to create a more comprehensive standard.  It is widely implemented in the United States by organisations in the healthcare space.  Unlike others, although there are external assessors that are involved in the certification process, HITRUST reviews all assessments and issues the certificate. Additionally, among all the frameworks above it tends to be the most expensive to implement.  It is important to choose a framework that matches your long-term security goals & needs.  At Accorian, we work with all of the above frameworks. We help organizations choose the right framework and aid with the implementation. This is done by augmenting our team into your security team to help steer the rollout, aid with query resolution, choosing of the right controls & workaround during mitigation advisory, facilitating the selection of vendors & products and end to end program management. 

View More

Ready to Start?

Security Compliance & Assessment Services

CONSULTING & ASSESSMENT SERVICES

PENETRATION TESTINGRed TEAM ASSESSMENTS
TECHNOLOGY STAFFING

Contact Info

E. info@accorian.com

T. +1-732-443-3468

Contact With Us

Office Address

Accorian Head Office

6,Alvin Ct, East Brunswick, NJ 08816 USA

Accorian Canada

Toronto

Accorian India

Ground Floor,11, Brigade Terraces, Cambridge Rd, Halasuru, Udani Layout, Bengaluru, Karnataka 560008, India

Ready to Start?​

Drop your CVs to joinourteam@accorian.com

Interested Position

Download Case study

Download SOC2 Guide