Data Operation
Upload data to a dataset
Users upload data to a dataset
- 1.Acquire data upload credentials
- 2.Transfer data directly to S3
- 3.Put callback
Acquire upload credentials
Request path
GET /v1/datasets/{id}/policies
Request parameter
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
Query
Name | Type | Descriptions | Value | ​ |
draftNumber | int | Yes | Draft number | ​ |
segmentName | string | No | Dataset segmentation | ​ |
expired | int | No | The token expiration time measured in seconds cannot be negative. The default is 60 seconds and any time greater than 300 seconds will be treated as 300 | ​ |
Request instance
curl --location --request GET '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/policies?expired=60&draftNumber=1' \
--header 'x-token: {your_accesskey}'\
--header 'Content-Type: application/json'
Output
# Response status
HttpStatus 200
# Response result
{
"result": {
"policy": "eyJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJmYXQtZGF0YXBsYXRmb3JtLXZlcnNpb24ifSxbInN0YXJ0cy13aXRoIiwiJGtleSIsIjNhZTBiYzI5M2Q3N2FiYjQwODZhMzc5ZjAyNjkwNmNlL2YwODk3NDUwLWNlZjEtNGQ3Zi05NDI3LWNiZGRhZjcxYzc1YS8iXSxbInN0YXJ0cy13aXRoIiwiJHN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyIsIiJdLFsic3RhcnRzLXdpdGgiLCIkQ29udGVudC1UeXBlIiwiIl0seyJ4LWFtei1jcmVkZW50aWFsIjoiQUtJQVFIVFdDQTNKTTZESTdVQ0YvMjAyMTAzMjQvdXMtd2VzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotZGF0ZSI6IjIwMjEwMzI0VDA1NDIwMFoifV0sImV4cGlyYXRpb24iOiIyMDIxLTAzLTI0VDA1OjQ0OjAwWiJ9",
"success_action_status": "200",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "AKIAQHTWCA3JM6DI7UCF/20210324/us-west-1/s3/aws4_request",
"x-amz-date": "20210324T054200Z",
"x-amz-signature": "8dbc309fa36caad5791907c385632e74a47362ca6f1517b748e150f1306851a0"
},
"extra": {
"backendType": "s3",
"host": "https://dataplatform-version.s3.us-west-1.amazonaws.com",
},
"expireAt": 1616564580
}
The obtained result can be deserialized directly and transferred directly to S3. extra host is the access address
Request path
POST The value of "host" within the "extra" array from the output above
Request parameter
body
Name | Type | Required | Descriptions |
x-amz-credential | string | Yes | Consistent with the returned x-amz-credential of the previously obtained credentials |
success_action_status | string | Yes | Consistent with the return value success_action_status of the previously obtained credentials |
x-amz-algorithm | string | Yes | Consistent with the returned x-amz-algorithm of the previously obtained credentials |
x-amz-signature | string | Yes | Consistent with the return value x-amz-signature of the previously obtained credentials |
policy | string | Yes | Consistent with the returned policy of the previously obtained credentials |
x-amz-date | string | Yes | Consistent with the return value x-amz-date of the previously obtained credentials |
key | string | Yes | Calculate the checksum of a file with SHA-1 cryptographic hash |
file | file | Yes | File |
Request instance
curl --location --request POST 'https://dataplatform-version.s3.us-west-1.amazonaws.com/' \
--form 'x-amz-credential=AKIAQHTWCA3JM6DI7UCF/20210323/us-west-1/s3/aws4_request' \
--form 'x-amz-algorithm=AWS4-HMAC-SHA256' \
--form 'x-amz-date=20210323T120601Z' \
--form 'x-amz-signature=42d7455003455a76796d67087b6a314b6c28445a14b80c9145029aeaa0091ec4' \
--form 'policy=eyJjb25kaXRpb25zIjpbeyJidWNrZXQiOiJmYXQtZGF0YXBsYXRmb3JtLXZlcnNpb24ifSxbInN0YXJ0cy13aXRoIiwiJGtleSIsIjNhZTBiYzI5M2Q3N2FiYjQwODZhMzc5ZjAyNjkwNmNlL2YwODk3NDUwLWNlZjEtNGQ3Zi05NDI3LWNiZGRhZjcxYzc1YS8iXSxbInN0YXJ0cy13aXRoIiwiJHN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyIsIiJdLFsic3RhcnRzLXdpdGgiLCIkQ29udGVudC1UeXBlIiwiIl0seyJ4LWFtei1jcmVkZW50aWFsIjoiQUtJQVFIVFdDQTNKTTZESTdVQ0YvMjAyMTAzMjMvdXMtd2VzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LHsieC1hbXotYWxnb3JpdGhtIjoiQVdTNC1ITUFDLVNIQTI1NiJ9LHsieC1hbXotZGF0ZSI6IjIwMjEwMzIzVDEyMDYwMVoifV0sImV4cGlyYXRpb24iOiIyMDIxLTAzLTIzVDEyOjA4OjAxWiJ9' \
--form 'success_action_status=200' \
--form 'key=b9676b075dff6d3333bbebb7cdd154f3631bb926' \
--form 'file=@/{localFile}'
Output
# Response status
HttpStatus 204
# Response result
{}
Upload complete callback
Request path
PUT /v1/datasets/{id}/callback
Request parameter
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
body
Name | Type | Required | Descriptions |
draftNumber | int | Yes | Draft number |
checksum | string | Yes | The checksum of an uploaded file. Calculate the checksum of a file with SHA-1 cryptographic hash |
remotePath | string | Yes | The remote file path, such as a.jpg |
segmentName | string | Yes | Segment name |
sensorName | string | No | Must be passed for the upload of fusion data; sensor name |
frameId | string | No | Must be provided for the upload of fusion data. The frameID should be in the format of ulid. Please refer to https://github.com/oklog/ulid​ |
timestamp | float | No | Not required to be provided for Fusion data upload |
label | map | No | For the annotation labels of the file, please refer to uploadLabel |
Request instance
curl --location --request PUT '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/callback' \
--header 'x-token: {your_accesskey}'\
--header 'Content-Type: application/json'\
--data-raw '{
"draftNumber": "1",
"checksum": "b9676b075dff6d3333bbebb7cdd154f3631bb926",
"remotePath": "car1.jpg",
"frameId": "01D78XYFJ1PRM1WPBCBT3VHMNV",
"segmentName": "car",
"sensorName": "camera_car",
"timestamp": 1532402934
}'
Output
# Response status
HttpStatus 200
# Response result
{}
Batch callback
Request path
PUT /v1/datasets/{id}/multi/callback
Request parameter
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
body
Name | Type | Required | Descriptions | ​ |
draftNumber | int | Yes | Draft number | ​ |
segmentName | string | Yes | Segment name | ​ |
objects | []object | Yes | The batch callback objects | ​ |
├─checksum | string | Yes | The checksum of an uploaded file. Calculate the checksum of a file with SHA-1 cryptographic hash | ​ |
├─remotePath | string | Yes | The remote file path, such as a.jpg | ​ |
├─sensorName | string | No | Must be passed for the upload of fusion data; sensor name | ​ |
├─frameId | string | No | Must be provided for the upload of fusion data. The frameID should be in the format of ulid. Please refer to https://github.com/oklog/ulid​ | ​ |
├─timestamp | float | No | Not required to be provided for Fusion data upload | ​ |
├─label | map | No | For the annotation labels of the file, please refer to uploadLabel | ​ |
Request instance
curl --location --request PUT '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/callback' \
--header 'x-token: {your_accesskey}'\
--header 'Content-Type: application/json'\
--data-raw '{
"draftNumber": "1",
"segmentName": "car",
"objects": [
{
"remotePath": "car1.jpg",
"checksum": "b9676b075dff6d3333bbebb7cdd154f3631bb926",
"frameId": "01D78XYFJ1PRM1WPBCBT3VHMNV",
"sensorName": "camera_car",
"timestamp": 1532402934,
"label":{}
}
]
}'
Output
# Response status
HttpStatus 200
# Response result
{}
Create a segment
POST /v1/datasets/{id}/segments
Request parameter
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
Body
Name | Type | Required | Descriptions |
draftNumber | int | Yes | Draft number |
name | string | Yes | Segment name. This cannot be null and should not include ".","" |
description | string | No | Segment description |
Request instance
curl --location --request POST '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/segments'\
--header 'x-token: {your_accesskey}'\
--header 'Content-Type: application/json' \
--data-raw '{
"draftNumber":1,
"name": "graviti",
"description": "this is a segment"
}'
Output
# Response status
HttpStatus 201
# Response result
{}
Delete segment
DELETE /v1/datasets/{id}/segments
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
Body
Name | Type | Required | Descriptions |
draftNumber | int | Yes | Draft number |
segmentName | string | Yes | segmentName |
Request instance
curl --location --request DELETE '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/segments' \
--header 'x-token: {your_accesskey}'\
--data-raw '{
"draftNumber":1,
"segmentName": "car",
}'
Output
# Response status
HttpStatus 200
# Response result
{}
Acquire dataset segment
GET /v1/datasets/{id}/segments
Request parameter
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
Query
Name | Type | Required | Descriptions |
draftNumber | int | No | Draft number. Only one of draft and commit should exist, but they should not exist at the same time |
commit | string | No | Commit ID, commit tag, or branch name |
limit | int | No | The default is set to 128. The maximum value for a single time is 128 |
offset | int | No | The default is 0 |
sortBy | string | No | Currently sorting is based or created_at and updated_at while entering other content will return InvalidParamsValue. |
orderBy | string | No | The default is desc (descending order). The valid content is asc (ascending order) and desc |
Request instance
curl --location --request GET '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/segments?commit=v1.0&offset=0&limit=2'\
--header 'x-token: {your_accesskey}'\
--header 'Content-Type: application/json'
Output
Normal dataset
# Response status
HttpStatus 200
{
"segments": [
{
"name": "car",
"description": "this is car"
},
{
"name": "graviti",
"description": "this is a segment"
}
],
"offset": 0,
"recordSize": 2,
"totalCount": 10
}
Upload the labelCatalog
PUT /v1/datasets/{id}/labels/catalogs
Request parameter
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
Body
Name | Type | Required | Descriptions |
draftNumber | int | Yes | Draft number |
catalog | map | yes | Annotation Type.
key:
CLASSIFICATION: annotation classification
BOX2D: 2D box annotation
BOX3D: 3D box annotation
POLYGON2D: 2D polygon annotation
POLYLINE2D: 2D polyline annotation
KEYPOINTS2D: 2D keypoints annotation
CUBOID2D: 2D cuboid annotation
SENTENCE: audio transcription annotation
For the annotation type above, each one of them has its own structure. You can refer to the request instances for detailed information. Please follow the format of the input value for each annotation type |
├─description | string | false | Descriptions |
├─categoryDelimiter | string | false | The delimiter used to separate the parent-child relationship of category. The default is not to support the parent-child relationship. The recommended delimiter is "." |
├─categories | object[] | false | Item type: object and category list, which includes all category information |
├─isTracking | bool | No | Tracking or not |
├─name | string | yes | Category name |
├─description | string | No | Category description |
├─attributes | object[] | No | Item type. It includes attribute-related information |
├─name | string | Yes | Attribute name |
├─type | string/string[] | false | attribute type:string, boolean, integer, number, array, null,such as"interger", ["integer", "null"] |
├─minimum | float | No | The minimum value of the attribute. When the attribute is numerical, this field defines the minimum value (closed interval) |
├─maximum | float | No | The maximum value of the attribute. When the attribute is numerical, this field defines the maximum value (closed interval) |
├─description | string | No | Attribute description |
├─enum | string[] | No | The options of the value of an attribute. This represents all the possible options this attribute may have. |
├─parentCategories | string[] | No | Relational categories represent the categories to which the attribute belongs to. If it is empty, then this is a global attribute |
├─items | object[] | No | Item type that defines multiple attributes. (When the "type" field is "array", it indicates multiple attributes) |
├─type | string/string[] | No | type of an attribute value with multiple options:string, boolean, integer, number, array, null,such as "interger", ["integer", "null"] |
├─minimum | float | No | The minimum value (closed interval) of the attribute value with multiple options |
├─maximum | float | No | The maximum value (closed interval) of the multiple attribute |
├─enum | string[] | No | The enumeration of the value of an attribute with multiple options. This represents all the possible options this attribute may have. |
Request instance
curl --location --request PUT '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/labels/catalogs'\
--header 'x-token: {your_accesskey}'\
--header 'Content-Type: application/json'\
--data-raw '{
"draftNumber": 1,
"BOX2D": {
"attributes": [
{
"description": "attribute 'traffic_light_color' is a multiple choice attribute",
"items": {
"enum": [
"unknown",
"red",
"yellow",
"green"
]
},
"name": "traffic_light_color",
"parentCategories": [
"traffic_light"
],
"type": "array"
},
{
"description": "attribute 'visibility' is a float value between 0 to 1",
"maximum": 1,
"minimum": 0,
"name": "visibility",
"type": "number"
},
{
"description": "this is color",
"enum": [
"red",
"blue",
"green"
],
"name": "color",
"parentCategories": [
"car"
],
"type": "string"
}
],
"categories": [
{
"description": "this is a car",
"name": "car"
}
],
"categoryDelimiter": ".",
"description": "this is a description of label catalog",
"isTracking": false
},
"BOX3D": {
"attributes": [
{
"description": "attribute 'traffic_light_color' is a multiple choice attribute",
"items": {
"enum": [
"unknown",
"red",
"yellow",
"green"
]
},
"name": "traffic_light_color",
"parentCategories": [
"traffic_light"
],
"type": "array"
},
{
"description": "attribute 'visibility' is a float value between 0 to 1",
"maximum": 1,
"minimum": 0,
"name": "visibility",
"type": "number"
},
{
"description": "this is color",
"enum": [
"red",
"blue",
"green"
],
"name": "color",
"parentCategories": [
"car"
],
"type": "string"
}
],
"categories": [
{
"description": "this is a car",
"name": "car"
}
],
"categoryDelimiter": ".",
"description": "this is a description of label catalog",
"isTracking": false
},
"CLASSIFICATION": {
"attributes": [
{
"description": "attribute 'traffic_light_color' is a multiple choice attribute",
"items": {
"enum": [
"unknown",
"red",
"yellow",
"green"
]
},
"name": "traffic_light_color",
"parentCategories": [
"traffic_light"
],
"type": "array"
},
{
"description": "attribute 'visibility' is a float value between 0 to 1",
"maximum": 1,
"minimum": 0,
"name": "visibility",
"type": "number"
},
{
"description": "this is color",
"enum": [
"red",
"blue",
"green"
],
"name": "color",
"parentCategories": [
"car"
],
"type": "string"
}
],
"categories": [
{
"description": "this is a car",
"name": "car"
}
],
"categoryDelimiter": ".",
"description": "this is a description of label catalog"
},
"CUBOID2D": {
"attributes": [
{
"description": "attribute 'traffic_light_color' is a multiple choice attribute",
"items": {
"enum": [
"unknown",
"red",
"yellow",
"green"
]
},
"name": "traffic_light_color",
"parentCategories": [
"traffic_light"
],
"type": "array"
},
{
"description": "attribute 'visibility' is a float value between 0 to 1",
"maximum": 1,
"minimum": 0,
"name": "visibility",
"type": "number"
},
{
"description": "this is color",
"enum": [
"red",
"blue",
"green"
],
"name": "color",
"parentCategories": [
"car"
],
"type": "string"
}
],
"categories": [
{
"description": "this is a car",
"name": "car"
}
],
"categoryDelimiter": ".",
"description": "this is a description of label catalog",
"isTracking": false
},
"KEYPOINTS2D": {
"attributes": [
{
"description": "attribute 'traffic_light_color' is a multiple choice attribute",
"items": {
"enum": [
"unknown",
"red",
"yellow",
"green"
]
},
"name": "traffic_light_color",
"parentCategories": [
"traffic_light"
],
"type": "array"
},
{
"description": "attribute 'visibility' is a float value between 0 to 1",
"maximum": 1,
"minimum": 0,
"name": "visibility",
"type": "number"
},
{
"description": "this is color",
"enum": [
"red",
"blue",
"green"
],
"name": "color",
"parentCategories": [
"car"
],
"type": "string"
}
],
"categories": [
{
"description": "this is a car",
"name": "car"
}
],
"categoryDelimiter": ".",
"description": "this is a description of label catalog",
"isTracking": false,
"keypoints": [
{
"description": "this is key point",
"names": [
"ear",
"leg",
"mouse"
],
"number": 3,
"parentCategories": [],
"skeleton": [
[
0,
1
],
[
1,
2
]
],
"visible": "1"
}
]
},
"POLYGON2D": {
"attributes": [
{
"description": "attribute 'traffic_light_color' is a multiple choice attribute",
"items": {
"enum": [
"unknown",
"red",
"yellow",
"green"
]
},
"name": "traffic_light_color",
"parentCategories": [
"traffic_light"
],
"type": "array"
},
{
"description": "attribute 'visibility' is a float value between 0 to 1",
"maximum": 1,
"minimum": 0,
"name": "visibility",
"type": "number"
},
{
"description": "this is color",
"enum": [
"red",
"blue",
"green"
],
"name": "color",
"parentCategories": [
"car"
],
"type": "string"
}
],
"categories": [
{
"description": "this is a car",
"name": "car"
}
],
"categoryDelimiter": ".",
"description": "this is a description of label catalog",
"isTracking": false
},
"POLYLINE2D": {
"attributes": [
{
"description": "attribute 'traffic_light_color' is a multiple choice attribute",
"items": {
"enum": [
"unknown",
"red",
"yellow",
"green"
]
},
"name": "traffic_light_color",
"parentCategories": [
"traffic_light"
],
"type": "array"
},
{
"description": "attribute 'visibility' is a float value between 0 to 1",
"maximum": 1,
"minimum": 0,
"name": "visibility",
"type": "number"
},
{
"description": "this is color",
"enum": [
"red",
"blue",
"green"
],
"name": "color",
"parentCategories": [
"car"
],
"type": "string"
}
],
"categories": [
{
"description": "this is a car",
"name": "car"
}
],
"categoryDelimiter": ".",
"description": "this is a description of label catalog",
"isTracking": false
},
"SENTENCE": {
"attributes": [
{
"description": "this is a description of attribute",
"enum": [
"red",
"blue",
"green"
],
"items": {
"enum": [
"red",
"blue",
"green"
],
"maximum": 10,
"minimum": 0,
"type": {}
},
"maximum": 10,
"minimum": 0,
"name": "color",
"parentCategories": [],
"type": {}
}
],
"description": "this is a description of label catalog",
"isSample": false,
"lexicon": "[[\"啊\", \"aa\", \"a1\"], [\"阿婆\": \"aa\", \"a1\", \"p\", \"o2\"], [\"爱\", \"aa\", \"ai4\"], [\"爱祖国\",\"aa\", \"ai4\", \"z\", \"u3\", \"g\", \"uo2\"]]",
"sampleRate": 32
}
}'
Output
HttpStatus 200
{}
Acquire dataset label
GET /v1/datasets/{id}/labels/catalogs
Request parameter
Path
Name | Descriptions | Value |
id | Yes | Dataset ID |
Query
Name | Type | Required | Descriptions |
draftNumber | int | No | Draft number. Only one of draft and commit should exist, but they should not exist at the same time |
commit | string | false | Commit ID, commit tag, or branch name |
Request instance
curl --location --request GET '{service}/v1/datasets/154e35bae8954f09969ef8c9445efd2c/labels/catalogs?commit=v1.0'\
--header 'x-token: {your_accesskey}'\
--header 'Content-Type: application/json'
Output
# Response status
HttpStatus 200
# Response result
{
"catalog": {
"BOX2D": {
"attributes": [
{
"name": "color"
}
],
"categories": [
{
"description": "hello",
"name": "car"
}
]
}
}
}