AIL - feeder from certificate transparency

This AIL feeder is a generic software to extract informations from certificate transparency

For the generation of domain name variations: ail-typo-squatting

Requirements

• redis
• cerstream
• dnspython
• ail-typo-squatting
• pyail
• beautifulsoup4
• M2Crypto

For M2Crypto:

sudo apt-get install libssl-dev swig python3-dev gcc

How to run

The program need to run two script to be functional, bin/ct.py and bin/feeder_ct.py.

The first one will publish ct informations on a redis db and the other one will subscribe to the channel and use any entry to compare with a list of variations of domain name. Redis pub/sub is used here.

If a variation match with an entry from redis db, then the variation is send to AIL to crawl the website.

Usage

dacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py --help  
usage: feeder_ct.py [-h] [-dn DOMAINNAME [DOMAINNAME ...]] [-fdn FILEDOMAINNAME] [-a] [-ats] [-ms] [-vt] [-w] [-o OUTPUT] [-v]

options:
  -h, --help            show this help message and exit
  -dn DOMAINNAME [DOMAINNAME ...], --domainName DOMAINNAME [DOMAINNAME ...]
                        list of domain name
  -fdn FILEDOMAINNAME, --filedomainName FILEDOMAINNAME
                        file containing list of domain name
  -a, --ail             Send domain to AIL crawler
  -ats, --ail_typo_squatting
                        Generate Variations for list pass in entry
  -ms, --matching_string
                        Match domain name if variations are in the domain name in any position
  -vt, --virustotal     Check domain on virus total
  -w, --warning			If CNAME is not the same as the matching domain then send a warning to Ail or just display a message.
  -o OUTPUT, --output OUTPUT
                        path to ouput location, default: ../output
  -v                    verbose, more display

Example of use

First of all ct.py need to be run.

Need to pass a text file, where each line is a variation of the original domain name. Variations can be generate at this repository: ail-typo-squatting

dacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py -fd circl.lu.txt

It's possible to generate variations directly in the program

dacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py -dn circl.lu -ats

It's possible to search variations in domain name instead of searching only if it's equal

dacru@dacru:~/git/ail-feeder-ct/bin$ python3 feeder_ct.py -dn circl.lu -ats -ms

JSON output format

the name of the JSON file will be the domains matching the variation.

if the dns resolving give no result, then the key "dns_resolve" will not be present in the JSON file.

{
	"certificat": "", 
    "domains": [], 
    "domain_matching": "", 
    "variation_matching": "", 
    "dns_resolve": {"A": [], "MX": [], "TXT": [], ...},
    "website_info": {"headers":{...}, "redirect": 1, "website_title": ""}
}

List of Resource Record use

NONE
A
NS
MD
MF
CNAME
SOA
MB
MG
MR
NULL
WKS
PTR
HINFO
MINFO
MX
TXT
RP
AFSDB
X25
ISDN
RT
NSAP
NSAP_PTR
SIG
KEY
PX
GPOS
AAAA
LOC
NXT
SRV
NAPTR
KX
CERT
A6
DNAME
OPT
APL
DS
SSHFP
IPSECKEY
RRSIG
NSEC
DNSKEY
DHCID
NSEC3
NSEC3PARAM
TLSA
HIP
CDS
CDNSKEY
CSYNC
SPF
UNSPEC
EUI48
EUI64
TKEY
TSIG
IXFR
AXFR
MAILB
MAILA
ANY
URI
CAA
TA
DLV