What is programmable search engine?

A programmable search engine is a free service offered by Google that allows you to personalize the power of Google’s search technology by including or excluding different types of sources. While the term ‘programmable’ might not be quite right, this option allows us to build a search engine that fits our needs. Programmable SE is not a different engine; it uses Google’s algorithms to enhance results from across the web.

One custom search engine can benefit our research in many ways. We can include up to 20 online sources, giving priority to results from those sources when delivering an outcome. The user can also give it a personal touch by adding certain functionality and appearance. For example, you can exclude - image search and web entirely, Google branding, and advertising. It should be mentioned that our SE will be limited in browsing results. Only the first ten pages will be displayed. However, this functionality will most likely be sufficient for most of us.

How Google’s Programmable SE could benefit our OSINT investigation?

As mentioned previously, a customized search engine could bring additional value to our research. First and foremost, it would accelerate the whole process. Instead of having to query site:example.com followed by ‘keyword’ for every possible source, we could now build a collection of sites and quickly search them together.

Second, relevance is key. The more specific domain and subdirectories you add, the more precise the results will be. For example, https://www.facebook.com/marketplace will only show results from this category and nothing more. Another positive feature of this product that is worth mentioning is you can add specific languages, excluding sites and keywords.

These are two reasons we, as OSINT investigators, need to include tailored SE in our toolset. However, I will stop here. In the next section, I hopefully will inspire you by showing you my process for building my collection of search engines.

So, let’s get started!

Create your own programmable search engine.

To create a new search engine, follow these steps:

  1. Log in to your Google account at https://programmablesearchengine.google.com/.
  2. You will be presented with a “Create a new search engine” panel. In “Search Settings”, enable “Image search” and “Safe search” if you want these options. Give your personalized SE a name. Then add the sites or pages you want to include. Lastly, check that you are human and click “Create”.

  1. Now that you have completed the initial setup, let’s do some tweaking. Return to the main page, and you will find an ‘Overview’ with several menu options - ‘Basic’, ‘Search Features’, ‘Look and Feel’, ‘Members’, ‘Ads’, and ‘Programmatic Access’. Click on the second option, ‘Search Features’ > ‘All search feature settings’ > ‘Refinements’ > ‘Add’. Here we should select all the sites we want to choose from.

This completes our initial setup. You can start using your customized search engine right away. However, I also like to include the following features: ‘autocomplete’, ‘green sky theme’, and ‘disable Google branding’.

Social Media Search Engine

Below is my Google social media search engine. I have included the most important social media sites: Facebook, Instagram, Twitter, LinkedIn, Tiktok, Pinterest, Snapchat, Reddit, and Tumblr. You can search the entire web or a specific website. I find an image search option(which you can enable/disable in the Settings menu) very useful. This type of SE is convenient for researching username handles or hashtags.

Document Search Engine

The second type of tailored search engine, which I consider crucial for our OSINT investigation, is ‘Document type SE’. I have added a slide-sharing type website that allows us to search a variety of documents - pdf and word; cvs, PowerPoint and photo files. To name a few - Slideshare, Issuu, Scribd and s3Amazon.

Video Search Engine

If I am interested in video content, I would probably go directly to YouTube. But what if some clips aren’t on the platform? Well, you are likely to miss them. Although YouTube is the biggest video platform out there, sites like Vimeo and Dailymotion can also add value to our research. It is always a good idea to check multiple sources.

In Conclusion

A programmable search engine is a handy tool with many advantages. I rely on them for almost all of my OSINT analyses. I personally build my SE toolset with a new sock puppet account every time I need it, and when I’m done, I don’t re-use it. If you are short on time, create your collection and bookmark it for future use.