# Introduction Write Code, Defend Democracy [Code for Democracy](https://codefordemocracy.org/) builds tools that make it easy to investigate relationships between campaign contributions, political narratives, and legislative outcomes in the US. This site contains detailed technical and methodological documentation that will help you get started with using and contributing to our platform. Begin by clicking on a topic in the sidebar or any of the sections below: {% content-ref url="/pages/-MUqlyFV6OzPSi4SwS00" %} [Tools](/tools/overview) {% endcontent-ref %} {% content-ref url="/pages/-MTIWGbfMB1gPVcdWVKc" %} [Data](/data/methodology) {% endcontent-ref %} {% hint style="info" %} If you're stuck, you can always send us an email at ! {% endhint %} # Overview ### Key Concepts Our tools were built to help you conduct recurring searches into datasets commonly utilized by political watchdogs with as few clicks and data manipulations as possible. To that end, we've created a few building blocks that power our workflows: * **Lists** — These are sets of entities (candidates, donors, terms, etc.) that can be defined manually using a set of IDs or dynamically using a combination of filters. Lists can be thought of as reusable collections of search terms that help you store things you look into frequently. * **Queries** — These are specific types of searches that utilize *recipes* (defined below) in conjunction with one or more *lists* and are the basis for visualizations and alerts. You'll want to save a query if you think you'll run it again when new data is available or if you want to share its results with others. * **Recipes** — These are analytical actions that take in one or more *lists* and in order to look at the data from our datasets in a pre-defined way and generate search results.As you use our tools, it's important to keep a few things in mind: ### Important Reminders * **Duplicates** — When summing amounts, especially in campaign finance data, be aware of records that might be duplicated due to transactions being reported by both the contributor and the recipient. We provide a data column that helps with this de-duplication by indicating which entity reported each transaction. * **Permissions** — While "hidden" lists or queries are not directly browsable by other users in our tools, their contents might still be exposed to other users if they are used by "public" queries or visualizations. As such, we recommend thinking of "hidden" objects truly as "hidden" and not as "private". {% hint style="info" %} Our tools are open source! You can contribute to the [Tools](https://github.com/codefordemocracy/tools) repository on GitHub and help us improve or add additional features. {% endhint %} # Workflows ### Main Workflows There are 4 main Code for Democracy workflows, which comprise the core of our search, visualization, and alerting tools: 1. [**Create a List**](https://tools.codefordemocracy.org/create/list/) — You can start here to build your own custom analysis or skip this workflow entirely if you just want to use the pre-loaded content. Our list creation workflow aims to make it easy to define groups of entities (candidates, donors, terms, etc.) to investigative. 2. [**Create a Query**](https://tools.codefordemocracy.org/create/query/) — Queries are the core of our bulk search capabilities. Each query uses a recipe, which is a specific search formula optimized for our datasets. By creating a query, you're creating a search that can be run again in the future with just one click. 3. [**Create a Visualization**](https://tools.codefordemocracy.org/create/visualization/) — This workflow takes the results from a query and formats it in customizable ways before sending it to a visualization engine. This workflow aims to make it easy to re-create visualizations when new data is available. 4. [**Create an Alert**](https://tools.codefordemocracy.org/create/alert/) — Alerts let you know when new results are available for a query, or perhaps when no results have been available for awhile. When an alert is triggered, our platform will send you an email with a link to the new data. ### Mini-Tools There are 3 Code for Democracy mini-tools, which combine exploratory with advanced discovery capabilities in order to aid in the creation of lists: 1. [**Explore Lists**](https://tools.codefordemocracy.org/explore/lists/) — A quick way to define new lists is to look at lists that have already been defined, and then clone them in order to add your own customizations. 2. [**Explore Relationships**](https://tools.codefordemocracy.org/explore/relationships/graph/) — This tool helps you explore connections between entities in our datasets in two different ways. It's meant to help brainstorm entities to add to your lists. 3. [**Explore Documents**](https://tools.codefordemocracy.org/explore/documents/) — Our topic lists are basically ontologies, and creating them can be complex. You can test how terms appear in documents using this tool. # Recipes The [Create a Query](https://tools.codefordemocracy.org/create/query/) workflow uses a variety of recipes to help you create bulk searches on our datasets. Recipes are grouped by tab into the following categories: * **Campaign Finance** — These recipes search contributions and expenditures from the FEC bulk data and contributions from the House and Senate lobbying disclosures. * **Lobbying Disclosures** — These recipes search the House and Senate lobbying disclosures. * **Tax Documents** — These recipes search the IRS 990, IRS 990EZ, and IRS 990PF filings. * **Political Ads & Narratives** — These recipes search Facebook's political ad library and front-page stories from over 3000 news sources. {% hint style="info" %} View our data [methodology](/data/methodology) page to read more about each of our data sources, their coverage, and their update cadence. {% endhint %} # Tutorials We've created a few step-by-step tutorials of example use cases to help you get started: * [Create an Alert for New Contributions to Candidates from California](https://scribehow.com/shared/Create-an-Alert-for-New__c38f6uJDSe2HCd3ierHfTg) * [Visualize Contributions from Senior Executives that were Refunded](https://scribehow.com/shared/Visualize-Contributions-from-Senior-Executives__JJyrITYSRpaOFLgjJXr2bA) * [Find Contributions from Lobbyists Lobbying on Healthcare](https://scribehow.com/shared/Find-Contributions-from-Lobbyists-Lobbying__XjI4aFnTTDy7XP1kLy4NiQ) * [Download Tax Documents that Mention Big Tech Companies](https://scribehow.com/shared/Download-Tax-Documents-that-Mention__c4joPvH1SE21TE1gFrNcEQ) * [Explore Donors Common to Manchin and Sinema](https://scribehow.com/shared/Explore-Donors-Common-to-Manchin__KRj1opGdQZejR-qGIa3VAQ) # Methodology This page documents the data sources, update frequencies, and processing methodology used to create Code for Democracy's data. {% hint style="info" %} You can also [view the status](https://api.codefordemocracy.org/view/status/) of each dataset to better understand its coverage and update cadence. {% endhint %} ### Government Data We believe that open data provided by the government is still the best place to start for any search. As such, these datasets are core to our platform: #### Campaign Finance Data from the [FEC's bulk data download](https://www.fec.gov/data/browse-data/?tab=bulk-data) is indexed daily, along with financial reports and more detailed Schedule A data from the [FEC API](https://api.open.fec.gov/developers/#/). We use the bulk data as the primary source for our campaign finance data because it is processed by the FEC before release, and therefore we consider it the cleanest source. However, in many cases, there may be a lag between when raw data is reported to the FEC and when the bulk data is available. #### Lobbying Disclosures Lobbying activity and related contribution reports are continuously indexed from both the [House](https://disclosurespreview.house.gov/) and [Senate](https://lda.senate.gov/system/public/) websites. For data from the House, we ingest the data by paging through the front-end website. For the data from the Senate, we ingest the data directly from their API. Therefore, it is very possible that data from the House is less comprehensive than data from the Senate. #### Tax Documents A subset of fields from the IRS 990, IRS 990EZ, and IRS 990PF filings are continuously indexed from the [AWS XML mirror](https://docs.opendata.aws/irs-990/readme.html). We use the provided index listings of available filings for each year in order to page through the individual XML filings. ### Narrative Data In additional to traditional open data sources, we also ingest a variety of datasets that are helpful for understanding the type of narratives occurring in political discourse: #### News News articles are indexed twice each day from the news sources rated by [Allsides](http://allsides.com/) and [Media Bias/Fact Check](https://mediabiasfactcheck.com/). Although we attempt to index all articles from each news source, in reality our coverage is should be thought of as a collection of "front-page" articles. #### Facebook Ads We also index all data related to "Issues, Elections or Politics" from the [Facebook Ads Library](https://www.facebook.com/ads/library/) on a continual basis. Our data comes from the Facebook API, and therefore it should be an exact mirror of the data available in the Ads Library. However, the universe of data available here is dependent on the accuracy of Facebook's own classification algorithms. #### Tweets We continuously index all tweets from a core group of Twitter users that are relevant political candidates, commentators, activists, fact checkers, or journalists. This is our least comprehensive dataset and is subject a multitude of potential misattribution and latency issues, so it should be used for exploratory purposes only. {% hint style="info" %} Our data pipeline is open source! See the [Data](https://github.com/codefordemocracy/data) repository on GitHub for details on how we are ingesting and processing each data source. {% endhint %} # API Our API is located at [api.codefordemocracy.org](https://api.codefordemocracy.org). Our API helps you get to the raw data behind our tools without needing to use the front-end interface. Get started by [viewing the available endpoints](https://api.codefordemocracy.org/view/endpoints/) or generating access credentials using the portal at [account.codefordemocracy.org](https://account.codefordemocracy.org). ### Code Samples You can get started with sending requests quickly using the following wrapper functions in python. Just fill in your `client_id` and `client_secret`. ```python from urllib.parse import urlencode import requests import json # set your credentials client_id = "XXXXXXXXXXXXXXXXXXXXXXX" client_secret = "XXXXXXXXXXXXXXXXXXXXXXX" # this function gets the response from our API def post(endpoint, body): url = "https://api.codefordemocracy.org" response = requests.post(url+endpoint, data=json.dumps(body), auth=(client_id, client_secret)) if response.status_code == 200: return json.loads(response.text) return [] ``` For example, you can use this code to print the `cmte_id` for `100` Lobbyist/Registrant PACs: ```python # set up your API call endpoint = "/graph/search/committees/" body = { "attributes": { "cmte_dsgn": "B" }, "pagination": { "limit": 100 } } # get the response elements = post(endpoint, body) # print the cmte_ids from the elements for element in elements: print(element["properties"]["cmte_id"]) ``` {% hint style="info" %} Our API is open source! See the [API](https://github.com/codefordemocracy/api) repository on GitHub and inspect the code for exact details on how we calculate each endpoint. {% endhint %}