ms/artifact-manager/README

   1 # CDS Artifact Manager
   2
   3 Artifact Manager is a very simple gRPC service that lets you upload, download and delete CBA archives. It can be ran as a standalone micro service (using `server.py`) or you can include it's methods in a service like `py-executor`.
   4
   5 ## Configuration
   6 Configuration is stored in `.ini` file, you can specify a path and name of a file using `CONFIGURATION` env variable.
   7 For possible variables please see example below (with inline comments):
   8 ```
   9 [artifactManagerServer]
  10 port=50052                    # A port on which the server will be listening
  11 logFile=server.log            # Path to a log file
  12 maxWorkers=20                 # Max number of concurent workers
  13 debug=true                    # Debug flag
  14 logConfig=logging.yaml        # A special MDC logger config
  15 fileRepositoryBasePath=/tmp/  # A FS path where we should store CBA files
  16 ```
  17
  18 ## Methods
  19 Below is a list of gRPC methods handled by the service. The `proto` files are available in `artifact-manager/manager/proto` directory.
  20
  21 All methods expect `CommonHeader` with:
  22 * `timestamp` - datetime in UTC with this: `%Y-%m-%dT%H:%M:%S.%fZ` format
  23 * `originatorId` - name of the component (eg. `CDS`)
  24 * `requestId` - ID of the request
  25 * `subRequestId` - Sub ID of the request
  26 * `flag` - TBD
  27
  28 and an `ActionIdentifiers` with following fields:
  29 * `blueprintName` - name of the blueprint
  30 * `blueprintVersion` - version number of blueprint (as string)
  31 * `actionName` - TBD
  32 * `mode` - TBD
  33
  34 ### Upload
  35
  36 Upload `CBA.zip` file for storage in artifact manager. File needs to be sent as a binary data in `fileChunk` field.
  37
  38 #### Example
  39
  40 ```
  41 stub: BlueprintManagementServiceStub = BlueprintManagementServiceStub(channel)
  42 with  open(file_path, "rb") as cba_file:
  43     msg: BlueprintUploadInput = BlueprintUploadInput()
  44     msg.actionIdentifiers.blueprintName =  "Test"
  45     msg.actionIdentifiers.blueprintVersion =  "0.0.1"
  46     msg.fileChunk.chunk = cba_file.read()
  47 return  stub.uploadBlueprint(msg)
  48 ```
  49
  50 ### Download
  51
  52 Download existing `CBA.zip` file.
  53
  54 #### Example
  55
  56 ```
  57 stub: BlueprintManagementServiceStub = BlueprintManagementServiceStub(channel)
  58 msg: BlueprintDownloadInput = BlueprintDownloadInput()
  59 msg.actionIdentifiers.blueprintName =  "Test"
  60 msg.actionIdentifiers.blueprintVersion =  "0.0.1"
  61 return  stub.downloadBlueprint(msg)
  62 ```
  63 ### Remove
  64
  65 Delete existing `CBA.zip` file.
  66
  67 #### Example
  68
  69 ```
  70 stub: BlueprintManagementServiceStub = BlueprintManagementServiceStub(channel)
  71 msg: BlueprintRemoveInput = BlueprintRemoveInput()
  72 msg.actionIdentifiers.blueprintName =  "Test"
  73 msg.actionIdentifiers.blueprintVersion =  "0.0.1"
  74 return  stub.removeBlueprint(msg)
  75 ```
  76
  77 ## Full gRPC Client Example
  78
  79 ```
  80 import logging
  81 import sys
  82 from argparse import ArgumentParser, FileType, Namespace
  83 from configparser import ConfigParser
  84 from datetime import datetime
  85 from pathlib import Path
  86
  87 import zipfile
  88
  89 from grpc import Channel, ChannelCredentials, insecure_channel, secure_channel, ssl_channel_credentials
  90
  91 from proto.BlueprintManagement_pb2 import (
  92     BlueprintDownloadInput,
  93     BlueprintRemoveInput,
  94     BlueprintUploadInput,
  95     BlueprintManagementOutput,
  96 )
  97 from proto.BlueprintManagement_pb2_grpc import BlueprintManagementServiceStub
  98
  99
 100 logging.basicConfig(level=logging.DEBUG)
 101
 102
 103 class ClientArgumentParser(ArgumentParser):
 104     """Client argument parser.
 105
 106     It has two arguments:
 107      - config_file - provide a path to configuration file. Default is ./configuration-local.ini
 108      - actions - list of actions to do by client. It have to be a list of given values: upload, download, remove.
 109     """
 110
 111     DEFAULT_CONFIG_PATH: str = str(Path(__file__).resolve().with_name("configuration-local.ini"))
 112
 113     def __init__(self, *args, **kwargs):
 114         """Initialize argument parser."""
 115         super().__init__(*args, **kwargs)
 116         self.description: str = "Artifact Manager client example"
 117
 118         self.add_argument(
 119             "--config_file",
 120             type=FileType("r"),
 121             default=self.DEFAULT_CONFIG_PATH,
 122             help="Path to the client configuration file. By default it's `configuration-local.ini` file from Artifact Manager directory",
 123         )
 124         self.add_argument(
 125             "--actions", nargs="+", default=["upload", "download", "remove"], choices=["upload", "download", "remove"]
 126         )
 127
 128
 129 class Client:
 130     """Client class.
 131
 132     Implements methods which can be called to server.
 133     """
 134
 135     def __init__(self, channel: Channel, config: ConfigParser) -> None:
 136         """Initialize client class.
 137
 138         :param channel: gprc channel object
 139         :param config: ConfigParser object with "client" section
 140         """
 141         self.channel: Channel = channel
 142         self.stub: BlueprintManagementServiceStub = BlueprintManagementServiceStub(self.channel)
 143         self.config = config
 144
 145     def upload(self) -> BlueprintManagementOutput:
 146         """Prepare upload message and send it to server."""
 147         logging.info("Call upload client method")
 148         with open(self.config.get("client", "cba_file"), "rb") as cba_file:
 149             msg: BlueprintUploadInput = BlueprintUploadInput()
 150             msg.actionIdentifiers.blueprintName = "Test"
 151             msg.actionIdentifiers.blueprintVersion = "0.0.1"
 152             msg.fileChunk.chunk = cba_file.read()
 153         return self.stub.uploadBlueprint(msg)
 154
 155     def download(self) -> BlueprintManagementOutput:
 156         """Prepare download message and send it to server."""
 157         logging.info("Call download client method")
 158         msg: BlueprintDownloadInput = BlueprintDownloadInput()
 159         msg.actionIdentifiers.blueprintName = "Test"
 160         msg.actionIdentifiers.blueprintVersion = "0.0.1"
 161         return self.stub.downloadBlueprint(msg)
 162
 163     def remove(self) -> BlueprintManagementOutput:
 164         """Prepare remove message and send it to server."""
 165         logging.info("Call remove client method")
 166         msg: BlueprintRemoveInput = BlueprintRemoveInput()
 167         msg.actionIdentifiers.blueprintName = "Test"
 168         msg.actionIdentifiers.blueprintVersion = "0.0.1"
 169         return self.stub.removeBlueprint(msg)
 170
 171
 172 if __name__ == "__main__":
 173     arg_parser: ClientArgumentParser = ClientArgumentParser()
 174     args: Namespace = arg_parser.parse_args()
 175
 176     config_parser: ConfigParser = ConfigParser()
 177     config_parser.read_file(args.config_file)
 178
 179     server_address: str = f"{config_parser.get('client', 'address')}:{config_parser.get('client', 'port')}"
 180     if config_parser.getboolean("client", "use_ssl", fallback=False):
 181         logging.info(f"Create secure connection on {server_address}")
 182         with open(config_parser.get("client", "private_key_file"), "rb") as private_key_file, open(
 183             config_parser.get("client", "certificate_chain_file"), "rb"
 184         ) as certificate_chain_file:
 185             ssl_credentials: ChannelCredentials = ssl_channel_credentials(
 186                 private_key=private_key_file.read(), certificate_chain=certificate_chain_file.read()
 187             )
 188         channel: Channel = secure_channel(server_address, ssl_credentials)
 189     else:
 190         logging.info(f"Create insecure connection on {server_address}")
 191         channel: Channel = insecure_channel(server_address)
 192
 193     with channel:
 194         client: Client = Client(channel, config_parser)
 195         for action in args.actions:
 196             logging.info("Get response")
 197             logging.info(getattr(client, action)())
 198
 199 ```