Wednesday, September 14, 2016

MongoDB/CRITs

Authenticated API
Mike Goffin edited this page on Jul 26 · 6 revisions

CRITs comes with an authenticated API. It is disabled by default and must be enabled in the Control Panel. Once it is enabled you must restart the web server for the URLs to be exposed. The API leverages Tastypie. A lot of the features of Tastypie are overridden by CRITs code for our needs. More information on Tastypie can be found here:
Users can visit their profile page and generate API keys for their necessary tasks. Each API key can be revoked if that key becomes compromised or the user no longer wishes to use it.
If you are using a web browser to access the API and you have an authenticated CRITs session already, you do not have to provide authentication information to utilize the API. To provide authentication information, append the following to your requests:
&username=<username>&api_key=<api_key>
Where <username> is your username, and <api_key> is the API key you wish to use.
API URLs
The URL structure for the API comes in this form:
/api/<version>/<resource>/...
The <version> comes in the form of v# where # is the API version you are using.
The <resource> is the content you wish to utilize within CRITs. Currently the available resources are:
  • actors
  • actoridentifiers
  • backdoors
  • campaigns
  • certificates
  • comments
  • domains
  • emails
  • events
  • exploits
  • indicators
  • ips
  • pcaps
  • raw_data
  • samples
  • targets
For example, if you wanted to get a list of campaigns, you would use:
/api/v1/campaigns/
This would return a list of campaigns in JSON format. If you wish to change the format to something else, you would append "?format=<format>" where <format> is the format you want the data returned to you in. We currently support the following formats:
  • json
  • yaml
  • xml
If you wish to get information about a single document, you would use:
/api/v1/campaigns/<object_id>/
If the results of the GET request have a file in GridFS, you can download them by appending "?file=1" on the end. By default the file(s) will be returned to you in raw format. If you wish to adjust which format the files are in when sent to you, you can append "&file_format=<format>" where <format> is one of 'base64', 'zlib', and 'invert'. The result will be a zip file with a password of 'infected' containing all of the files found with your GET request. If you query for just a single document you will still get a zip file with the same password. If there is an issue generating the zip file (whether due to no files being found, or some other error) you will get a non-zip file which contains any issues.
Queries against the API are done via GET requests. If you wish to upload content to the system through the API, you must use a POST.
Searching Using GET Parameters
If you wish to limit the results of your GET request by the value of one or more fields, you can do so by adding more parameters. The format is similar to how you use search operators in the global search box. Here's an example:
/api/v1/samples/?c-campaign.name=Bad Guys&c-filename=foo.txt
You'll notice a couple things here. The first is that there is a "c-" in front of the field name. This tells the API that this parameter should be used as a field to search against to limit your results. In our example, only samples with a Campaign Name of "Bad Guys" AND a filename of "foo.txt" will be returned to us. Finally, in this format the fields are both full matches.
The above assumes the parameters are to be used in an AND query. If you wish to instead use your parameters in an OR query, you can add "or=1" to your parameter list:
/api/v1/samples/?or=1&c-campaign.name=Bad Guys&c-filename=foo.txt
This will return samples with a Campaign Name of "Bad Guys" OR a filename of "foo.txt".
As long as you have a "c-" in front of the parameter, you can search against any field in the database, whether it is a CRITs supported field or not!
If you wish to use regex, you can add "&regex=1". This will convert all "c-" parameters to regex searches (except those with comparison operators which is explained below).
/api/v1/samples/?c-campaign.name=Bad&c-filename=foo&regex=1
This will find all Samples where the filename contains "foo" and the Campaign Name contains "Bad".
If you wish to use the MongoDB comparison operators ('gt', 'gte', 'lt', 'lte', 'in', 'nin'), you can use the syntax "__<operator>" when defining your field. Some examples:
/api/v1/samples/?c-size__lte=100
/api/v1/samples/?c-bucket_list__in=foo,bar
Note:
Using a comparison operator will override "&regex=1" so none of the fields which use comparison operators will be converted into regex.
Limiting GET Request Results
If you wish to limit the fields in the results of your GET request, you can use two features:
only:
Only populate these and any required fields.
exclude:
Do not populate these fields if they are not required.
For example:
/api/v1/samples/?only=filename,mimetype,md5
Will only populate those and any required fields for each document. The inclusion of required fields is intentional. The classes expect that these fields are populated if they are required and do not have a default value to set, so if you do not include them they will complain.
You can combine both only and exclude. The result will be all of the fields from only as long as they aren't in the exclude list, combined with any required fields.
Examples:
cURL:
curl -F "filedata=@/path/to/file.pcap" -F "source=<your source>" http://localhost:1337/api/v1/pcaps/?username=<your username>&api_key=<your api key>
Python:
Upload a PCAP file
import requests
files = { 'filedata': open('/path/to/file.pcap', 'rb') }
data = {
'api_key': '<your api key>',
'username': '<your username>',
'source': '<source name>'
}
r = requests.post(url, data=data, files=files)
Adding a Domain
import requests
data = {
'api_key': '<your api key>',
'username': '<your username>',
'source': '<source name>',
'domain': 'www.fakedomain.evil',
'ip': '127.0.0.2'
}
r = requests.post(url, data=data)
if r.status_code == 201:
print "Successfully added "+params['domain']
Listing the Domains:
import requests
params = {
'api_key': '<your api key>',
'username': '<your username>',
}
r = requests.get(url, params=params, verify=False)
r.json
Ruby:
Adding a Domain
require 'net/http'
params = {
'username' => '<your username>',
'api_key' => '<your api key>',
'source' => '<source name>',
'domain'=>'ruby.evil.org'
}
res = Net::HTTP.post_form(uri,params)
if res.code == "201" {
print "Successfully added "+params['domain']
}
Listing the Domains
require 'net/http'
require 'json'
params = {
'username' => '<your username>',
'api_key' => '<your api key>'
}
uri.query = URI.encode_www_form(params)
res=Net::HTTP.get_response(uri)
result=JSON.load(res.body)
POST Responses
In most cases when submitting a POST to add new content to CRITs you will get a response in the following JSON format:
{
"return_code": <code>,
"type": <type>,
"id": <object_id>,
"message": <message>,
"url": <url>
}
The return_code is usually 0 for success, 1 for failure. Some API calls may wish to extend this to include other codes to represent other results. Those return codes should be listed in the appropriate sections below.
In the event a new TLO was created or content was successfully added to an existing TLO, the TLO type and ID will be returned to you. In addition, a URL will be provided which can be used to query the API for the details of that TLO.
If there is a message to give context, whether the request was successful or not, it will also be provided.
In some situations the entire response may not look like this. Refer to the sections below for any special conditions you may need to look out for.
Common POST Parameters
These parameters are frequently used across most top-level objects in CRITs.
bucket_list:
Comma-separated list of buckets.
source:
Name of the source which provided this information.
  • should be a source already in your CRITs instance.
method:
Method in which the information was acquired from the source.
reference:
Reference for the source of the information.
campaign:
Campaign associated with this information.
confidence:
Campaign confidence associated with this information.
  • must be one of "low", "medium", or "high"
ticket:
Associated Ticket.
Actor API
GET:
/api/v1/actors/
/api/v1/actors/<object_id>/
POST:
/api/v1/actors/
Unique POST Parameters:
name:
Name of the Actor.
aliases:
Comma-separated list of actor aliases.
description:
Description of the Actor.
Actor Identifier API
GET:
/api/v1/actoridentifiers/
/api/v1/actoridentifiers/<object_id>/
POST:
/api/v1/actoridentifiers/
Unique POST Parameters:
identifier_type:
Type of the Actor Identifier.
identifier:
Value of the Actor Identifier.
Backdoor API
GET:
/api/v1/backdoors/
/api/v1/backdoors/<object_id>/
POST:
/api/v1/backdoors/
Unique POST Parameters:
name:
Name of the Backdoor.
version:
Version of the Backdoor
aliases:
Comma-separated list of backdoor aliases.
description:
Description of the Backdoor.
Campaign API
GET:
/api/v1/campaigns/
/api/v1/campaigns/<object_id>/
POST:
/api/v1/campaigns/
Unique POST Parameters:
aliases:
Comma-separated list of campaign aliases.
description:
Description of the campaign.
name:
Name of the campaign.
Certificate API
GET:
/api/v1/certificates/
/api/v1/certificates/<object_id>/
POST:
/api/v1/certificates/
Unique POST Parameters:
description:
Description of the certificate.
filedata:
The certificate.
related_id:
ObjectId of the related top-level object.
related_md5:
MD5 of the related top-level object if it has one.
related_type:
The CRITs type of the related top-level object.
relationship:
The type of relationship.
Comment API
POST:
/api/v1/comments/
Unique POST Parameters:
comment:
The Comment you would like to add.
object_id:
The ObjectId of the item to add comment to.
object_type:
The Object Type of this object.
Domain API
GET:
/api/v1/domains/
/api/v1/domains/<object_id>/
POST:
/api/v1/domains/
Unique POST Parameters:
add_indicators:
If you wish to add Indicators to the system for this information.
add_ip:
Tell the API that you are also adding an IP related to this domain.
domain:
The domain name.
ip:
The IP you want to relate to this domain.
ip_type:
The type of the IP ('IPv4 Address' is most common).
ip_source:
The source of the IP.
ip_method:
The method in which you acquired this IP from the source.
ip_reference:
The reference about this IP source.
same_source:
If you want the IP to have the same source as the domain.
Email API
GET:
/api/v1/emails/
/api/v1/emails/<object_id>/
POST:
/api/v1/emails/
Unique POST Parameters:
upload_type:
One of "eml", "msg", "raw", "yaml", "fields".
filedata:
The email in EML, Raw, or YAML format.
email_id:
The ObjectId of the email if it is in YAML format and exists already.
password:
The password for the attachment if this is an MSG file.
If you are uploading a type of "fields", you can include these parameters but only "date" is required:
date:
The Date field.
to:
The To field.
cc:
The CC field.
from_address:
The From field.
sender:
The Sender field.
subject:
The email subject.
reply_to:
The Reply-To field.
helo:
The HELO.
boundary:
The Boundary.
message_id:
The email Message-ID
raw_header:
The raw header of the email.
raw_body:
The raw body of the email.
originating_ip:
The Originating IP field.
x_originating_ip:
The X-Originating-IP field.
x_mailer:
The X-Mailer.
Event API
GET:
/api/v1/events/
/api/v1/events/<object_id>/
POST:
/api/v1/events/
Unique POST Parameters:
description:
Description of the event.
event_type:
Type of Event.
date:
Date of the event.
title:
Title of the event.
Exploit API
GET:
/api/v1/exploits/
/api/v1/exploits/<object_id>/
POST:
/api/v1/exploits/
Unique POST Parameters:
name:
Name of the Exploit.
cve:
CVE identifier of the Exploit.
description:
Description of the Exploit.
Indicator API
GET:
/api/v1/indicators/
/api/v1/indicators/<object_id>/
POST:
/api/v1/indicators/
Unique POST Parameters:
add_domain:
Add a Domain to CRITs if this is a domain indicator.
add_relationship:
Add a relationship if this is a Domain, IP, etc.
indicator_confidence:
The confidence level of this Indicator.
  • Must be one of "unknown", "benign", "low", "medium", "high".
indicator_impact:
The impact level of this Indicator.
  • Must be one of "unknown", "benign", "low", "medium", "high".
type:
The type of the Indicator.
threat_type:
The threat type of the Indicator.
attack_type:
The attack type of the Indicator.
value:
Indicator value.
description:
A description of the Indicator.
Indicator Activity API
POST:
/api/v1/indicator_activity/
Unique POST Parameters:
object_id:
The ObjectId of the Indicator to add activity to.
start_date:
The start datetime of the activity.
end_date:
The end datetime of the activity.
description:
A description of the activity.
IP API
GET:
/api/v1/ips/
/api/v1/ips/<object_id>/
POST:
/api/v1/ips/
Unique POST Parameters:
description:
Add a description to the IP address
add_indicator:
Add this IP as an Indicator.
indicator_reference:
Reference for the source of the Indicator.
ip:
IP Address.
ip_type:
Type of IP Address.
PCAP API
GET:
/api/v1/pcaps/
/api/v1/pcaps/<object_id>/
POST:
/api/v1/pcaps/
Unique POST Parameters:
filedata:
The PCAP.
related_id:
ObjectId of the related top-level object.
related_md5:
MD5 of the related top-level object if it has one.
related_type:
The CRITs type of the related top-level object.
relationship:
The type of relationship.
Raw Data API
GET:
/api/v1/raw_data/
/api/v1/raw_data/<object_id>/
POST:
/api/v1/raw_data/
Unique POST Parameters:
upload_type:
One of "metadata" or "file".
copy_relationships:
Copy the relationships of the linked raw data version.
  • Requires a link_id
data:
The raw data if the upload type is "metadata".
description:
Description of the raw data.
filedata:
The raw data if the upload type is "file".
data_type:
The type of raw data. Must match choices in the database.
link_id:
The Link ID of an existing version of this raw data.
title:
Title for the raw data.
tool_details:
Details about the tool.
tool_name:
The tool, utility, or host of the raw data.
tool_version:
The version of the tool, utility, or host of the raw data.
Sample API
GET:
/api/v1/samples/
/api/v1/samples/<object_id>/
POST:
/api/v1/samples/
Unique POST Parameters:
upload_type:
One of "metadata" or "file".
description:
The description of the file.
filename:
The name of the file if this is a "metadata" upload.
md5:
The MD5 of the file if this is a "metadata" upload.
sha1:
The SHA1 of the file if this is a "metadata" upload.
sha256:
The SHA256 of the file if this is a "metadata" upload.
size:
The size of the file if this is a "metadata" upload.
mimetype:
The mimetype of the file if this is a "metadata" upload.
filedata:
The sample if the upload type is "file".
password:
The password to unzip the file if it is zipped and the upload type is "file".
file_format:
The format of the file if the upload type is "file".
  • Must be one of "zip", "rar", or "raw".
related_md5:
The MD5 of the related sample if this was an embedded sample.
related_id:
The ObjectId of the related TLO.
related_type:
The type of TLO to relate to.
Screenshot API
GET:
/api/v1/screenshots/
/api/v1/screenshots/<object_id>/
POST:
/api/v1/screenshots/
Unique POST Parameters:
upload_type:
One of "ids" or "screenshot".
filedata:
The screenshot if the upload type is "screenshot".
screenshot_ids:
A comma-separated list of existing screenshot ObjectIds to add to the top-level object.
tags:
A comma-separated list of tags for the screenshot.
oid:
The ObjectId of the top-level object the screenshot(s) is/are for.
otype:
The type of the top-level object the screenshot(s) is/are for.
description:
Description of the screenshot.
Service API
POST:
/api/v1/services/
Unique POST Parameters:
object_type:
The type of top-level object you are updating.
object_id:
The ObjectId to search for.
analysis_id:
The ID of the analysis task you want to update.
result:
The 'result' contents of your analysis result.
result_type:
The 'Type' of your result.
result_subtype:
The 'subtype' of your result.
result_is_batch:
Set to "1" if you are pushing multiple results.
log_message:
The contents of the log message.
log_level:
The level of the log (defaults to 'info').
status:
The status to set the task to when it is finished.
  • Must be one of "error" or "completed"
finish:
Set to "1" to note that this task is finished and to mark it "complete".
If you include the "result" field, you will be required to also provide the "result_type" and "result_subtype". Also, if you include the "log_message" field, "log_level" will be set to "info" if it is not provided.
If you want to submit multiple results at once you have to set "result_is_batch" to "1" and supply result, result_type, and result_subtype as JSON list. Make sure that all three lists have the same length so there is a type and subtype for each result.
Signature API
GET:
/api/v1/signatures/
/api/v1/signatures/<object_id>/
POST:
/api/v1/signatures/
Unique POST Parameters:
copy_relationships:
Copy the relationships of the linked signature version.
  • Requires a link_id
data:
The signature that will be added.
description:
Description of the signature.
data_type:
The type of signature. Must match choices in the database.
link_id:
The Link ID of an existing version of this signature.
title:
Title for the signature.
Target API
GET:
/api/v1/targets/
/api/v1/targets/<object_id>/
POST:
/api/v1/targets/
Unique POST Parameters:
firstname:
Target first name.
lastname:
Target last name.
division:
The division of an organization the Target is a part of.
department:
The department of an organization the Target is a part of.
email_address:
The email address of the Target.
organization_id:
The ID of the organization if they have one.
title:
The job title of the Target.
note:
Notes about the Target.
Updating TLOs through the API using PATCH
Over time we are beginning to expose handler functions to the API allowing you to make modifications to TLOs using them. This requires a PATCH request. We will not document the specific parameters to include as they will change along with the handler they map to. For information on what parameters to include, look at the handlers and provide the appropriate arguments. A couple notes, however:
id_:
Anywhere you notice an id_ argument to a handler, we provide it for you.
type_:
Anywhere you notice a type_ argument to a handler, we provide it for you.
user:
Anywhere you notice a "user" argument to a handler, we provide it for you.
Attempts at providing those values yourself will be dropped and replaced by the values we provide. For example, if you wanted to add releasability to a Sample, you don't need to provide a 'Sample' type nor the ObjectId of the sample since we can determine that from your PATCH request. PATCH requests are submitted to the details URL for a TLO like so:
/api/v1/samples/<object_id>/?action=add_releasability
You would then look at what the "add_releasability" handler requires for arguments, and provide them in your request.
Here is a list of currently exposed handlers:
Submitting a PATCH request with python requests
As usual, this can be accomplished with the python requests library. As an example, let's use the ticket_add action on an Event TLO.
import datetime
import json
import requests
event_id = crits_event_object_id
submit_url = 'https://crits_server.com/api/v1/events/{}/'.format(event_id)
params = {
    'api_key' : api_key,
    'username' : username,
}
headers = {
    'Content-Type' : 'application/json',
}
# date must be in the format %Y-%m-%d %H:%M:%S.%f
formatted_date = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
data = {
    'action' : 'ticket_add',
    'ticket' : {
        'ticket_number' : 12345,
        'date' : formatted_date,
    }
}
r = requests.patch(submit_url, headers=headers, proxies=proxies, params=params,
                data=json.dumps(data), verify=False)
if r.status_code == 200:
    print('Yay!')
There are a couple of important things to note about the above example. First, all datetime objects must be formatted properly with the provided strftime string. Second, when passing a python dict structure as part of the data, you must pass it through json.dumps() so it is handled properly by the requests library. Otherwise the corresponding function on the CRITs side will receive incorrect data.
Finally, we can view the ticket_add function within the CRITs code itself to verify the data it expects in the PATCH request.
def ticket_add(type_, id_, ticket, user, **kwargs):
    :param ticket: The ticket to add.
    :type ticket: dict with keys "date", and "ticket_number".
    :param user: The user creating the ticket.
    :type user: str
    :returns: dict with keys:
            "success" (boolean),
            "object" (str) if successful,
            "message" (str) if failed.
    """
    # Code
Recall that the type_, id_, and user are all passed in automatically by the API handler. Notice the 'ticket' param is a dict with two keys, so our example code used json.dumps() above.
Common PATCH actions for all TLOs
action_add:
action_update:
action_remove:
add_object:
add_releasability:
description_update:
forge_relationship:
run_service:
source_add_update:
source_remove:
status_update:
ticket_add:
ticket_update:
ticket_remove:
Actor PATCH actions
update_actor_tags:
attribute_actor_identifier:
set_identifier_confidence:
remove_attribution:
set_actor_name:
update_actor_aliases:
Indicator PATCH actions
activity_add:
activity_remove:
activity_update:
ci_update:
set_indicator_attack_type:
set_indicator_threat_type:
Signature PATCH actions
update_dependency:
update_max_version:
update_min_version:
update_signature_data:
update_signature_type:
update_title:


Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)