-# Search Engine Micro Service
+# Search Data Service Micro Service
-The _Search Engine_ micro service exposes APIs via REST which allow clients to interact with the search database back end without requiring direct knowledge of or interaction with the underlying technology.
-
-## High Level Concepts
-This section establishes some of the terminology and concepts that relate to interacting with the _Search Engine_ service.
-A much more detailed examination of these concepts can be found on the [Search Engine Design Share](http://d2athenaconf:8090/confluence/display/AAI/AAI-4633%3A+Search+DB+Abstraction%3A+Expose+REST+Interface) Confluence page.
+[](https://bestpractices.coreinfrastructure.org/projects/1737)
+
+## Overview
+The _Search Data Service_ acts as an abstraction layer for clients which have a need to interact with data which is most suitably persisted in a searchable document store. The purpose of imposing an abstraction service between the client and the document store itself is to decouple clients from any direct knowledge of any specific document store technology, allowing the underlying technology to be swapped out without a direct impact to any clients which interact with search data.
+
+Please refer to the following sub-sections for more detailed information:
+
+[High Level Concepts](./CONCEPTS.md) - Discussion of the high level concepts and building blocks of the _Search Data Service_.
+
+[Index API](./INDEXES.md) - Details regarding manipulating document indexes.
-### Documents
-_Documents_ are the _things_ that we want to put into our document store. At its most basic, a _document_ is a collection of _fields_ which contain the data that we want to be able to store and query.
+[Document API](./DOCUMENTS.md) - Details regarding manipulating documents.
-_Fields_ are defined as having a name, a type, and optional parameters indicating whether or not the field is intended to be searchable, and if so, how it should be indexed.
+[Search API](./SEARCH.md) - Details regarding querying the data set.
+
+[Bulk API](./BULK.md) - Details regarding submitted bulk operation requests.
-### Indexes
-An _index_ is essentially a collection of _documents_. It is the top-level container into which we will store our documents. A single data store may have multiple _indexes_ for the purposes of segregating different types of data (most queries are performed across all documents within *single* instance).
----
## Getting Started
### Building The Micro Service
-After checking out the project, execute the following Maven command from the project's top level directory:
+After cloning the project, execute the following Maven command from the project's top level directory to build the project:
> mvn clean install
+
+Now, you can build your Docker image:
+
+ > docker build -t openecomp/search-data-service target
-### Running The Micro Service Locally
-To run the microservice in your local environment, execute the following Maven command from the project's top level directory:
+### Deploying The Micro Service
+
+Push the Docker image that you have built to your Docker repository and pull it down to the location that you will be running the search service from.
+
+Note that the current version of the _Search Data Service_ uses _ElasticSearch_ as its document store back end. You must therefore deploy an instance of ElasticSearch and make it accessible to the _Search Data Service_.
- > mvn -P runAjsc
+**Create the following directories on the host machine:**
-### Running The Micro Service Within An Eclipse Environment
-It is often extremely useful to be able to run a micro service from within Eclipse in order to set breakpoints and perform general debugging activities.
+ /logs
+ /opt/app/search-data-service/appconfig
+
+You will be mounting these as data volumes when you start the Docker container.
-For a good reference on how to launch any of the D2 micro services from within an Eclipse environment, refer to the following Confluence page: [Running An AJSC Container Within Eclipse](http://d2athenaconf:8090/confluence/pages/viewpage.action?pageId=1840887#DevelopingMicroserviceswithAT&T-RunninganAJSCContainerwithinEclipse)
+**Populate these directories as follows:**
----
+##### Contents of /opt/app/search-data-service/appconfig
-## Public Interfaces
+The following files must be present in this directory on the host machine:
-### Echo Service
-The _Search Database Abstraction_ micro service supports the standard echo service to allow it to be 'pinged' to verify that the service is up and responding.
+_analysis-config.json_
+Create this file with exactly the contents below:
-The echo service is reachable via the following REST end point:
- http://{host}:9509/services/search-data-service/v1/jaxrsExample/jaxrs-services/echo/{input}
+ [
+ {
+ "name": "whitespace_analyzer",
+ "description": "A standard whitespace analyzer.",
+ "behaviours": [
+ "Tokenize the text using white space characters as delimeters.",
+ "Convert all characters to lower case.",
+ "Convert all alphanumeric and symbolic Unicode characters above the first 127 ASCII characters into their ASCII equivalents."
+ ],
+ "tokenizer": "whitespace",
+ "filters": [
+ "lowercase",
+ "asciifolding"
+ ]
+ },
+ {
+ "name": "ngram_analyzer",
+ "description": "An analyzer which performs ngram filtering on the data stream.",
+ "behaviours": [
+ "Tokenize the text using white space characters as delimeters.",
+ "Convert all characters to lower case.",
+ "Convert all alphanumeric and symbolic Unicode characters above the first 127 ASCII characters into their ASCII equivalents.",
+ "Apply ngram filtering using the following values for minimum and maximum size in codepoints of a single n-gram: minimum = 1, maximum = 2."
+ ],
+ "tokenizer": "whitespace",
+ "filters": [
+ "lowercase",
+ "asciifolding",
+ "ngram_filter"
+ ]
+ }
+ ]
-### Indexes
-The _Search Engine_ service supports simple creation and deletion of document indexes via the following REST API calls:
+_filter-config.json:_
-##### Create Index
- Method : POST
- URL : https://<host>:9509/services/search-data-service/v1/search/indexes/<index>/
- URL Params : index - The name of the index to be created.
- Request Payload:
- A document structure expressed as json.
-
- Response Payload:
- {"url": "< resource location of the index >"
+Create this file with exactly the contents below:
-##### Delete Index
- Method : DELETE
- URL : http://<host>:9509/services/search-data-service/v1/search/indexes/<index>/
- URL Params : index - The name of the index to be deleted.
- Request Payload:
- None
-
-
-### Documents
-
-##### Create Document Without Specifying a Document Identifier
-Documents can be created via a POST request with no document identifier specified. In this case the document store will generate an identifier to associate with the document.
-
- Method : POST
- URL : https://<host>:9509/services/search-data-service/v1/search/indexes/<index>/documents/
- URL Params : index - The name of the index to create the document in.
- Request Payload:
- Document contents expressed as a JSON object containing key/value pairs.
-
- Response Payload:
- { "etag": "string", "url": "string" }
-
-##### Create or Update Document With a Specified Document Identifier
-Documents can also be created via a PUT request which includes an identifier to associate with the document. The put endpoint is actually used for both creates and updates, where this is distinguished as follows:
-* If the request header DOES NOT include a value in the If-Match field, then the request is assumed to be a document create.
-* If the request header DOES contain a value in the If-Match field, then the request is assumed to be a document update.
-
- Method : PUT
- URL : https://<host>:9509/services/search-data-service/v1/search/indexes/<index>/documents<document id>
- URL Params : index - The name of the index to create or update the document in.
- document id - The identifier of the document to be created or updated.
- Request Payload:
- Document contents expressed as a JSON object containing key/value pairs.
-
- Response Payload:
- { "etag": "string", "url": "string"}
-
-##### Delete a Document
-
- Method: : DELETE
- URL : https://<host>:9509/services/search-data-service/v1/search/indexes/<index>/documents<document id>
- URL Params : index - The name of the index to remove the document from.
- document id - the identifier of the document to be deleted.
- Request Payload:
- None.
-
-##### Retrieve a Document
-
- Method: : GET
- URL : https://<host>:9509/services/search-data-service/v1/search/indexes/<index>/documents<document id>
- URL Params : index - The name of the index to retrieve the document from.
- document id - the identifier of the document to be retrieved.
- Request Payload:
- None.
-
-
-### Searching the Document Store
-Search statements are passed to the _Search Data Service_ as a JSON object which is structured as follows:
-
-_Filters_
-* A "filter" stanza defines a set of queries to be run in _non-scoring-mode_ to reduce the document set to a smaller subset to be searched.
-* The filter stanza is optional - omitting it implies that the query is _unfiltered_.
-* This stanza is represented as a JSON object with the following structure:
-
- "filter": {
- "all": [ { query }, { query },....{ query }],
- "any": [ { query }, { query },....{ query }]
- },
-
-Where:
-* the _all_ list defines a set of queryies such that ALL queries in the list must be satisfied for the document to pass the filter.
-* the _any_ list defines a set of queryies such that ANY single query in the list must be satisfied for the document to pass the filter.
-
-_Queries_
-The following types of query statements are supported by the _Search Data Service_:
-
-_Term Query_:
-
-A term query attempts to match the literal value of a field, with no advanced parsing or analysis of the query string. This type of query is most appropriate for structured data like numbers, dates and enums, rather than full text fields.
-
- // Find documents where the specified field contains the supplied value
- "match": {
- "field": "value"
- }
-
- // Find documents where the specified field DOES NOT contain the supplied value
- "not-match": {
- "field": "value"
- }
+ [
+ {
+ "name": "ngram_filter",
+ "description": "Custom NGram Filter.",
+ "configuration": " \"type\": \"nGram\", \"min_gram\": 1, \"max_gram\": 50, \"token_chars\": [ \"letter\", \"digit\", \"punctuation\", \"symbol\" ]"
+ }
+ ]
-_Parsed Query_:
+_elastic-search.properties_
-Parsed queries apply a query parser to the supplied query string in order to determine the exact query to apply to the specified field.
-The query string is parsed into a series of terms and operators, as described below:
+This properties file configures the _Search Data Service_ for communication with ElasticSearch.
+The contents of this file will be determined by your ElasticSearch deployment:
-Terms may be any of the following:
-* single words
-* exact phrases, as denoted by enclosing the phrase in open and close quotations. Example: "this is my exact phrase"
-* regular expressions, as denoted by wrapping the expressing in forward slash ( / ) character. Example: /joh?n(ath[oa]n)/
+ es.cluster-name=<<name of your ElasticSearch cluster>>
+ es.ip-address=<<IP address of your ElasticSearch instance>>
+ es.http-port=9200
+ # Optional parameters
+ es.uri-scheme=<<either http or https>>
+ es.trust-store=<<key store containing the certs of trusted servers>>
+ es.trust-store-password=<<encrypted passsword to open the trust store>>
+ es.key-store=<<key store containing the client cert>>
+ es.key-store-password=<<encrypted passsword to open the client key store>>
+ es.auth-user=<<username for HTTP Basic Authentication>>
+ es.auth-password=<<encrypted passsword for HTTP Basic Authentication>>
-The supported operators are as follows:
-* AND - Both terms to the left or right of the operator MUST be present
-* OR - Either the term to the left or right of the operator MUST be present
-* NOT - The term to the right of the operator MUST NOT be present.
- "parsed-query": {
- "field": "fieldname",
- "query-string": "string"
- }
-
-_Range Query_:
-
- Range queries match fields whose term value falls within the specified numeric or date range.
- Supported bounds operators include:
- * gt - Greater than
- * gte - Greater than or equal to
- * lt - Less than
- * lte - Less than or equal to
-
- "range": {
- "field": "fieldname",
- "operator": "value",
- "operator": "value"
- }
-
-##### Examples
-The following snippet illustrates a search statement describing a filtered query which uses examples of all of the supported query types:
+##### Contents of the /opt/app/search-data-service/app-config/auth Directory
+
+The following files must be present in this directory on the host machine:
+
+_search\_policy.json_
+
+Create a policy file defining the roles and users that will be allowed to access the _Search Data Service_. This is a JSON format file which will look something like the following example:
{
- "filter": {
- "all": [{"range": {"field": "timestamp", "lte": "2016-12-01T00:00:00.558+03:00"}}],
- "any": [ ]
- },
-
- "queries": [
- {"match": {"field": "name", "value": "Bob"}},
- {"parsed-query": {"field": "street-name", "query-string": "Main OR First"}},
- {"range": {"field": "street-number", "gt": 10, "lt": 50}}
+ "roles": [
+ {
+ "name": "admin",
+ "functions": [
+ {
+ "name": "search", "methods": [ { "name": "GET" },{ "name": "DELETE" }, { "name": "PUT" }, { "name": "POST" } ]
+ }
+ ],
+ "users": [
+ {
+ "username": "CN=searchadmin, OU=My Organization Unit, O=, L=Sometown, ST=SomeProvince, C=CA"
+ }
+ ]
+ }
]
}
-##### REST Endpoint
+_tomcat\_keystore_
- Method: : POST
- URL : https://<host>:9509/services/search-data-service/v1/search/indexes/<index>/query
- URL Params : index - The name of the index to apply the query to.
+Create a keystore with this name containing whatever CA certificates that you want your instance of the _Search Data Service_ to accept for HTTPS traffic.
- Request Payload:
- {
- "filter": {
- "all": [ { query }, { query },....{ query }],
- "any": [ { query }, { query },....{ query }]
- },
-
- "queries": [
- { query },
- .
- .
- { query }
- ]
- }
+**Start the service:**
-### Bulk Operations
-Bulk operations allow the client to bundle a number of actions into a single REST request.
-It is important to note that individual operations bundled into a bulk request are considered by the _Search Service_ to be completely independent operations. This has a few important consequences:
-* No guarantees are made with respect to the order in which the individual operations will be processed by the document store.
-* There is no implied transactionality between the operations. Individual operations my succeed or fail independently of one another, and it is entirely possible for the client to receive back a result set indicating a mix of success and failure results for the individual operations.
-
-##### Submit Bulk Request
- Method : POST
- URL : http://<host>:9509/services/search-data-service/v1/search/bulk/
- URL Params : NONE
- Request Payload:
- A json structure containing all of the bundled actions to be performed.
- It must correspond to the following format:
- [
- { "operation": {{<metaData>}, {<document>},
- { "operation": {{<metaData>}, {<document>},
- .
- .
- { "operation": {{<metaData>}, {<document>},
- ]
-
- Where,
- operation - Is one of: "create", "update", or "delete"
-
- metaData - A structure containing meta-data associated with the individual operation to be performed. Valid fields include:
- "url" - The resource identifier of the document to be operated on.
- "etag" - Identifies the version of the document to be acted on. Required for "update" and "delete" operations.
-
- document - The document contents for "create" and "update" operations.
-
- Example Payload:
- [
- {"create": {"metaData": {"url": "/services/search-data-service/v1/indexes/the-index/documents/1"}, "document": {"f1": "v1", "f2": "v2"}}},
- {"create": {"metaData": {"url": "/services/search-data-service/indexes/the-index/documents/2"}, "document": {"f1": "v1", "f2": "v2"}}},
- {"update": {"metaData": {"url": "/services/search-data-service/v1/search/indexes/the-index/documents/8", "etag": "1"}, "document": {"f1": "v1a", "f2": "v2a"}}},
- {"delete": {"metaData": {"url": "/services/search-data-service/v1/search/indexes/the-index/documents/99", "etag": "3"}}}
- ]
-
- Response Payload:
- The response body will contain an aggregation of the collective results as well as separate status codes for each of the operations in the request.
- Example:
- {
- "total_operations": 4,
- "total_success": 1,
- "total_fails": 3,
- "results": [
- {"operation": "create", "url": "/services/search-data-service/v1/search/indexes/the-index/documents/1", "etag": "1", "status-code": "201", "status-message": "OK"},
- {"operation": "create", "url": "/services/search-data-service/v1/search/indexes/the-index/documents/2", "etag": "1", "status-code": "201", "status-message": "OK"},
- {"operation": "update", "url": "/services/search-data-service/v1/search/indexes/the-index/documents/8", "etag": "2", "status-code": "200", "status-message": "OK"},
- {"operation": "delete", "url": "/services/search-data-service/v1/search/indexes/the-index/documents/2", "status-code": "200", "status-message": "OK"}
- ]
-}
\ No newline at end of file
+You can now start the Docker container for the _Search Data Service_, in the following manner:
+
+ docker run -d \
+ -p 9509:9509 \
+ -e CONFIG_HOME=/opt/app/search-data-service/config/ \
+ -e KEY_STORE_PASSWORD={{obfuscated password}} \
+ -e KEY_MANAGER_PASSWORD=OBF:{{obfuscated password}} \
+ -v /logs:/opt/aai/logroot/AAI-SDB \
+ -v /opt/app/search-data-service/appconfig:/opt/app/search-data-service/config \
+ --name search-data-service \
+ {{your docker repo}}/search-data-service
+
+Where,
+
+ {{your docker repo}} = The Docker repository you have published your Search Data Service image to.
+ {{obfuscated password}} = The password for your key store/key manager after running it through the Jetty obfuscation tool.
+
+
+
+
\ No newline at end of file
Code Review / aai / search-data-service.git / blobdiff