python-3.x – Advanced Python decorators

Question:

Introduction

To come up with an answer to a question on StackOverflow in Spanish that, unfortunately, was finally deleted by the OP, I spent a while researching Python decorator programming and in particular one that was flexible enough to adapt to a particularly complex situation.

Although the question was deleted, I think that what I learned elaborating the answer may be useful to someone else and I find it a shame that it is lost, so I rephrase the question here, slightly modified so as not to harm the OP who asked the original question and the erased.

It is about implementing a decorator that can be used so that the decorated function can validate whether or not it has received certain parameters in a dictionary.

For example, this would be the function that the dictionary receives:

def funcion_ejemplo(data={}, prefijo="<div>", sufijo="</div>"):
  return f"{prefijo}{data['item']} x {data['cantidad']}{sufijo}"

Which could be invoked for example like this:

cadena = funcion_ejemplo({'item': 'tornillo', 'cantidad': 6})

and it would produce the string "<div>tornillo x 6</div>" .

Naturally, if this function is not passed the correct dictionary (that is, one that does not have the "item" field or the "cantidad" field, the function will break. In a general case it could even give incorrect results. We want to add to the function code that verifies if it has received the correct data and otherwise generates a ValueError exception. For this particular example, this validation would be done like this:

  # ...
  for campo in ["item", "cantidad"]:
       if campo not in data or not data[campo]:  # (el campo existe pero viene vacío)
          raise(ValueError(f"Falta el campo {campo} en el diccionario"))

It would now be a matter of delegating that work to a decorator that we can call required_data so that it can be used like this:

@required_data("item", "cantidad")
def funcion_ejemplo(data={}, prefijo="<div>", sufijo="</div>"):
  return f"{prefijo}{data['item']} x {data['cantidad']}{sufijo}"

How would you program such a decorator?

Answer:

The decorator that is requested is very complex for several reasons:

  • It is a decorator that receives parameters. This in itself already adds an important conceptual complexity, because the decorator must define an internal function that, when executed, returns a decorator, so that internal function must in turn define another internal function, which will be the final decorator. .
  • The number of parameters to be received by the decorator is variable. In the example there are two: "item", "cantidad" , but it can be assumed that it can receive any other quantity of fields to validate.
  • The function to be decorated has default values ​​among its parameters, which complicates the implementation of the wrapper .

Making a good, sturdy, generic decorator that can cope with all of this is tricky and the result necessarily ugly and quite unreadable. So we go by steps.

Simplistic initial release

Let's suppose for now that whenever the function is called once it is decorated, the data parameter will be passed to it. This frees us from the problem that it can be invoked like this simply with funcion_ejemplo() , which for now will facilitate the implementation of the decorator.

In this case, the decorator you are looking for could be as follows (I have included comments to try to explain – and explain myself – the complexity):

def required_data(*args):

  # Este es el wrapper que recibe como parámetro la función a decorar
  def _wrapper(f):
    # Dentro hay otro decorador
    # Este es el código que se ejecutará al llamar a la funcion decorada
    def verify_and_call(*inner_args):
      # *inner_args son los parámetros
      req_data = args          # Tomamos del closure ("item", "cantidad")
      data = inner_args[0]     # Sacar el primer parámetro que se le pasa a f
      # Realizar la validación
      for d in req_data:
        if d not in data or not data[d]:
          raise(ValueError(f"Falta el campo '{d}' en el diccionario"))

      # Si la validación pasa, podemos invocar a la función f y pasarle los
      # mismos parámetros, retornando el resultado que devuelva f
      return f(*inner_args)

    # El _wrapper retorna la función que acabamos de definir
    return verify_and_call

  # El decorador retorna el _wrapper
  return _wrapper

Example of use:

@required_data("item", "cantidad")
def funcion_ejemplo(data={}, prefijo="<div>", sufijo="</div>"):
  "La función puede tener una documentación en forma de docstring"
  return f"{prefijo}{data['item']} x {data['cantidad']}{sufijo}"

Demonstration that it works:

>>> item1 = {"item": "tornillo", "cantidad": 6}
>>> item2 = {"item": "tuerca" }  # Falta cantidad, es erróneo
>>> funcion_ejemplo(item1)
'<div>tornillo x 6</div>'
>>> funcion_ejemplo(item2)
Traceback (most recent call last)
[...]
ValueError: Falta el campo 'cantidad' en el diccionario

However, if we don't pass parameters to it, it fails in a weird way:

>>> funcion_ejemplo()
Traceback (most recent call last)
[...]
---> 10       data = inner_args[0]     # Sacar el parámetro que se le pasa a f

IndexError: tuple index out of range

Indeed, by not passing the *inner_args parameter, it arrives empty, and the attempt to access its element [0] produces an exception.

On the other hand, it also doesn't work well if we pass the parameter by name:

>>> funcion_ejemplo(data=item1)
Traceback (most recent call last)
----> 1 funcion_ejemplo(data=person1)

TypeError: verify_and_call() got an unexpected keyword argument 'data'

Indeed what happens now is that having defined verify_and_call() so that it receives only positional parameters ( *inner_args ), it does not let us pass named parameters to it.

We could think of fixing it by adding another **kwargs parameter, but we would still be in trouble because in any case the data parameter would not be in inner_args[0] as the function supposes.

Second version, handling positional or named parameters

If we add to verify_and_call() the possibility of receiving **kwargs , then data may be in *inner_args if it was passed by position, or in **kwargs if it was passed by name. So the code should search both sites:

def required_data(*args):

  # Este es el wrapper que recibe como parámetro la función a decorar
  def _wrapper(f):
    # Dentro hay otro decorador
    # Este es el código que se ejecutará al llamar a la funcion decorada
    def verify_and_call(*inner_args, **kwargs):
      # *inner_args son los parámetros
      req_data = args          # Tomamos del closure ("item", "cantidad")
      if "data" in kwargs:
        data = kwargs["data"]
      elif len(inner_args)>0:
        data = inner_args[0]
      else:
        raise(ValueError(f"Falta el parámetro data"))
      # Realizar la validación
      for d in req_data:
        if d not in data or not data[d]:
          raise(ValueError(f"Falta el campo '{d}' en el diccionario"))

      # Si la validación pasa, podemos invocar a la función f y pasarle los
      # mismos parámetros, retornando el resultado que devuelva f
      return f(*inner_args, **kwargs)

    # El _wrapper retorna la función que acabamos de decfinir
    return verify_and_call

  # El decorador retorna el _wrapper
  return _wrapper

Notice how it looks if "data" is in kwargs and if it is not trying to take from inner_args and if it is not an exception is raised. Also notice how the call to f() is also passed **kwargs in case that is where the data came from.

Now it works with named or positional parameters:

>>> funcion_ejemplo(item1)
'<div>tornillo x 6</div>'
>>> funcion_ejemplo(data=item1)
'<div>tornillo x 6</div>'

but it keeps crashing if we don't pass parameters (although this time it's because we purposely raised an exception signaling it):

>>> funcion_ejemplo()
Traceback (most recent call last)
[...]
ValueError: Falta el parámetro data

This is much better, but it prevents us from decorating a function that receives a default parameter, like this:

@required_data("item", "cantidad")
def funcion_ejemplo(data={"item": "noname", "cantidad": 1}, prefijo="<div>", sufijo="</div>"):
  "La función puede tener una documentación en forma de docstring"
  return f"{prefijo}{data['item']} x {data['cantidad']}{sufijo}"

Here we are passing as a default value to data a dictionary that should validate correctly since it has "item" and "cantidad" , but in reality the validation does not even take place if we call funcion_ejemplo() without parameters, because our decorator expects a parameter called data and we are not passing it on.

There is another minor detail, and it is that wrapping the function in this way loses the docstring of the original function:

>>> help(funcion_ejemplo)
Help on function verify_and_call in module __main__:

verify_and_call(*inner_args, **kwargs)

Instead of giving us the help on the function function_example funcion_ejemplo() is showing us that of its wrapper, verify_and_call()

Latest version, defaults and docstrings

To solve the problem of the docstring we will use functools.wraps() which is used to copy the prototype and documentation of the decorated function to the decorator function.

In order to access the default value of data in the decorated function we inspect.signature use inspect.signature .

This is the final version of the decorator:

from functools import wraps
from inspect import signature

# Este es el decorador "externo". 
# args es la lista de parámetros que se le pasa al decorador
# por ejemplo ("name", "last_name")
def required_data(*args):

  # Este es el wrapper que recibe como parámetro la función a decorar
  def _wrapper(f):
    sig = signature(f)   # Obtener la declaración y lista de parámetros de f
    @wraps(f)     # Esto es para preservar nombre y docstring de la función decorada
      # Este es el código que se ejecutará al llamar a la funcion decorada
    def verify_and_call(*inner_args, **kwargs):
      # inner_args y kwargs son los parámetros con los que se invocará a la
      # función decorada
      req_data = args           # Tomamos del closure ("item", "cantidad")
      if inner_args:            # Sacamos el parámetro "data" de la función decorada
        data = inner_args[0]    # Pudo pasarsele como el primer argumento posicional
      elif "data" in kwargs:    # o como un argumento por nombre
        data = kwargs["data"]
      elif "data" in sig.parameters:    # o estar en la lista de valores por defecto de f
        data = sig.parameters["data"].default
      else:                     # o no estar en ningún sitio
        raise(ValueError(f"Falta el parámetro data"))
      # Realizar la validación
      for d in req_data:
        if d not in data or not data[d]:
          raise(ValueError(f"Falta el campo '{d}' en el diccionario"))

      # Si la validación pasa, podemos invocar a la función f y pasarle los
      # mismos parámetros, retornando el resultado que devuelva f
      return f(*inner_args, **kwargs)

    # El _wrapper retorna la función que acabamos de decfinir
    return verify_and_call

  # El decorador retorna el _wrapper
  return _wrapper

It is used exactly the same as the previous ones:

@required_data("item", "cantidad")
def funcion_ejemplo(data={}, prefijo="<div>", sufijo="</div>"):
  "La función puede tener una documentación en forma de docstring"
  return f"{prefijo}{data['item']} x {data['cantidad']}{sufijo}"

and we can verify that it already supports all the desired characteristics:

>>> # No le paso parámetro, rompe porque el parámetro por defecto es {} que no tiene "item"
>>> funcion_ejemplo()
[...]
ValueError: Falta el campo 'item' en el diccionario


>>> # Parámetro por posicion y por nombre
>>> funcion_ejemplo(data=item1)
'<div>tornillo x 6</div>'
>>> funcion_ejemplo(item1)
'<div>tornillo x 6</div>'


>>> # La ayuda funciona
>>> help(funcion_ejemplo)
Help on function funcion_ejemplo in module __main__:

funcion_ejemplo(data={}, prefijo='<div>', sufijo='</div>')
    La función puede tener una documentación en forma de docstring

It also works correctly if we decorate a function that has a valid default value in data :

@required_data("item", "cantidad")
def funcion_ejemplo(data={"item": "noname", "cantidad": 1}, prefijo="<div>", sufijo="</div>"):
  "La función puede tener una documentación en forma de docstring"
  return f"{prefijo}{data['item']} x {data['cantidad']}{sufijo}"


>>> funcion_ejemplo()
'<div>noname x 1</div>'
Scroll to Top